-- Logs begin at Thu 2022-08-25 04:20:58 UTC, end at Thu 2022-08-25 05:41:38 UTC. -- Aug 25 04:51:55.063382 dv-cinder-853910-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 25 04:51:55.071846 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Aug 25 04:51:55 2022] *** Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: nodename: dv-cinder-853910-2-iscsi Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: machine: x86_64 Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: clock source: unix Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: pcre jit disabled Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: detected number of CPU cores: 4 Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: current working directory: / Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: detected binary path: /usr/bin/uwsgi-core Aug 25 04:51:55.075600 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: your processes number limit is 47680 Aug 25 04:51:55.076215 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: your memory page size is 4096 bytes Aug 25 04:51:55.076215 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: detected max file descriptor number: 2048 Aug 25 04:51:55.076215 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: lock engine: pthread robust mutexes Aug 25 04:51:55.076215 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: thunder lock: enabled Aug 25 04:51:55.076215 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 25 04:51:55.076215 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 25 04:51:55.115834 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: Python main interpreter initialized at 0x562b0ce2e9c0 Aug 25 04:51:55.115834 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: python threads support enabled Aug 25 04:51:55.115834 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: your server socket listen backlog is limited to 100 connections Aug 25 04:51:55.115834 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: your mercy for graceful operations on workers is 90 seconds Aug 25 04:51:55.116156 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: mapped 403077 bytes (393 KB) for 2 cores Aug 25 04:51:55.116266 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: *** Operational MODE: preforking *** Aug 25 04:51:55.116529 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: *** uWSGI is running in multiple interpreter mode *** Aug 25 04:51:55.116529 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: spawned uWSGI master process (pid: 81749) Aug 25 04:51:55.116730 dv-cinder-853910-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 25 04:51:55.118876 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: spawned uWSGI worker 1 (pid: 81762, cores: 1) Aug 25 04:51:55.118876 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: spawned uWSGI worker 2 (pid: 81763, cores: 1) Aug 25 04:51:55.118876 dv-cinder-853910-2-iscsi devstack@keystone.service[81749]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 25 04:51:59.409225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 25 04:51:59.409225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: from cryptography.utils import int_from_bytes Aug 25 04:51:59.410029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 25 04:51:59.410029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: from cryptography.utils import int_from_bytes Aug 25 04:51:59.410148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 25 04:51:59.410148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: from cryptography.utils import int_from_bytes Aug 25 04:51:59.410148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 25 04:51:59.410148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: from cryptography.utils import int_from_bytes Aug 25 04:51:59.459523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 25 04:51:59.459706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 25 04:51:59.459819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 25 04:51:59.459920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] command line args: [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 25 04:51:59.459999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 25 04:51:59.460142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 25 04:51:59.460225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] command line args: [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 25 04:51:59.460330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ================================================================================ {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 25 04:51:59.460451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 25 04:51:59.460579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] admin_token = **** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.460664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ================================================================================ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 25 04:51:59.460821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] config_dir = [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.460900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] admin_token = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] config_dir = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] config_source = [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] debug = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] config_source = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] debug = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.461992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.462317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.462386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.462543 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.462649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.462758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.462860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.462961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] list_limit = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] log_config_append = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463370 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_config_append = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] log_dir = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_dir = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] log_file = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.463979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_file = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.464924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.465901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.466055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.466179 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.466285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.466368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.466515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.466602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.466783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.466861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.467037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.467125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.467312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.467523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.467705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] publish_errors = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.467856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] publish_errors = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.468975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] standard_threads = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] standard_threads = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] use_journal = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.469978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] use_json = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_journal = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] use_stderr = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_json = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] use_syslog = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_stderr = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] use_syslog = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 25 04:51:59.470812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.470980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.471063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.471220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.471296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.471457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.471540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.471705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.471791 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.471942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.472970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.473138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.473220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.473368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.473444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.473622 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.473703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.473861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.473937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.474095 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.474170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.474337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.474417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.474571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.474655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.474805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.474889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.475060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.475137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.475301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.475377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.475536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.475621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.475769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.475850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.476963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.477997 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.478150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.478232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.478380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.478465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.478609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.478686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.478856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.478932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.479087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.479171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.479325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.479407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.479553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.479658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.479790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.479905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.480926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.481955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.482096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.482182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.482325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.482401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.482581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.482662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.482820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.482900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.483982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.484990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.485145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.485221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.485373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.485463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.485619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.485700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.485877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.485956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.486107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.486186 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.486335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.486413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.486567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.486643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.486802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.486885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.487990 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.488068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.488224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.488301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.488459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.488536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.488692 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.488769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.488927 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.489987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.490129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.490214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.490363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.490441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.490594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.490679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.490825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.490903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.491060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.491138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.491327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.491492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.491580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.491651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.491780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.491864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.492949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.493960 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.494112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.494201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.494341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.494418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.494576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.494652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.494809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.494894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.495046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.495123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.495284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.495361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.495540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.495626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.495778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.495855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] auth.external = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.external = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.496977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] auth.password = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.497064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.497207 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] auth.token = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.497293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.password = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.497435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] auth.token = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.497513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.497691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.497812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.497926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.498515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.498679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.498757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.498920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.498998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.499162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.499295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.499376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.499449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.499611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] credential.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.499689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.499847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.499933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.500092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.500170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.500337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.500415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.500576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.500662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.500817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.500900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.501055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.501133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.501289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.501368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.501539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.501642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.501803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.501880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.502050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.502136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.502305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.502384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.502560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.502637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.502821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.502900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.503063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.503141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.503311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.503394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.503541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] federation.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.503620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.503776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.503855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.504955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.505039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.505195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.505272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.505426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.505503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.505695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.505778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.505929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.506963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.507118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.507197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.507351 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.507429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.507584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.507663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.507811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.507907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.508988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.509066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.509217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.509295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.509450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.509528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.509716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.509797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.509963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.510043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.510211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.510290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.510454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.510538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.510703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.510827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.510952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.511967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.512105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.512183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.512341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.512418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.512578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.512667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.512819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.512928 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.513046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.513125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.513289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.513375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.513530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.514263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.514394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.514481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.514585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.514674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.514833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.514912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.515065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.515143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.515300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.515385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.515542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.515643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.515886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.515963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.516126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.516214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.516364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.516443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.516601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.516678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.516838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.516923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.517083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.517162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.517328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.517406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.517563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.517703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.517819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.517899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.518050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.518128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.518302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.518389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.518535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.518613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.518771 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.518849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.519940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520186 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.520992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.521147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.521233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.521374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.521453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.521625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.521707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.521862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.521952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.522091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.522169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.522319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.522397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.522566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.522656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.522808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.522885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523604 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.523997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.524086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.524230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.524309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.524462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.524539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.524703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.524826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.524924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.525980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.526132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.526213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.526364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.526441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.526596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.526675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.526830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.526909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.527070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.527148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.527308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.527386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.527538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.527617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.527768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.527846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] resource.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.528953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.529032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.529199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.529276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.529471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.529661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.529758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.529967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.530120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.530240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.530473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.530552 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.530708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] role.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.530848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.530957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] role.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.531191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] role.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.531271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.531439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.531581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] role.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.531687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.531876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.531955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.532129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.532208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.532378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.532459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.532626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.532705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.532872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.532950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.533117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.533197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.533347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.533428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.533611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.533693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.533865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.533945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.534094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.534172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.534340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.534418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.534585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.534733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.534831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.534970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.535074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.535219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.535468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.535546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.535705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.535783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.535953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.536996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.537155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.537233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.537387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.537473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.537644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.537726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.537877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.537955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.538109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.538199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.538340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.538419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.538590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.538668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.538823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.538912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.539980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] token.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.540065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.540213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.540300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.540457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.540536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.540691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.540770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.540933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.541992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.542147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.542226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.542378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.542464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.542614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.542694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.542847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.542925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.543090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.543178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.543353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.543433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.543579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.543666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.543821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.543900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.544074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.544160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.544312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.544391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.connection = **** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.544552 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.544632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.544787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.544937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.545047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.545125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.545279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.545357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.545511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.545616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.545778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.545857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.546947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.547954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.548034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81762) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 25 04:51:59.548107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.548326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.548560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 25 04:51:59.548729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81763) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 25 04:51:59.578338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.578567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.578649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.578790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.578890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.579064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.587486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.587624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.615979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.616165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.616254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.616331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.675992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.676166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.727135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.727315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.727716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.890072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.890311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.890460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.890550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.890695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.890779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.890917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.891008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.891133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.891215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.891350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.891567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.914199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.914433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.914666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.914894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.915114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.915329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.915543 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.916522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.916751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.916973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.917195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.917414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.917652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.917870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.927400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.928518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.940783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.940962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.941054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.941131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.941205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.945414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.945499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.945567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.945708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.945791 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.945879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.945956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.946029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81763) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.946091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81762) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 25 04:51:59.961171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.961287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.961372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 25 04:51:59.961452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 25 04:51:59.961741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.961827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.961944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 25 04:51:59.962298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 25 04:51:59.962484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.962564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.962748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.962828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.963262 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.963351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.963524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.963604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.973057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.973155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.974051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.974138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.974202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.974275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.974347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.974419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.980922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:51:59.981011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:51:59.984339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:51:59.984428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:51:59.984625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.984720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.984912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.984995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.985199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.985281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.985486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.985568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.985774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.985857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.986046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.986127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.986320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.986401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.986704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.986792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.986985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.987073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.987256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:51:59.987335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.009935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.010216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.014197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.014197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.014350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.014350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.016943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.017189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.017276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.017480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.017564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.017794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.023898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.024185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.024270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.024508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.028308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.028602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.028698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.028887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.028978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.029169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.029895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.030024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.030109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.030303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.030386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.030574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.030656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.030842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.030932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.031112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.047978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.048258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.048426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.048678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.052171 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.052600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.055406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.055699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.059881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.059881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.060020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.060399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.063120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.063600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.066404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.066710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.067002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.067086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.067298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.067381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.067588 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.067672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.067876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.067960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.068181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.068471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.084287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.094381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.094381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.094534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.094614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.094679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.094679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.094679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.095020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.095101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.095174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.095174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.095326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.095326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.096784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.097071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.097375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.097572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.097697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.097883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.097966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.098184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.098473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.098836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.109876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.110158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.110441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.110723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.110827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.111008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.111152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.111285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.111368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.111572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.111655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.111855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.112144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.112434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.127169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.127979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.130234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.131170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.133130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.134101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.134602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.135548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.136088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.136374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.136658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.137083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.137368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.137679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.143234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.144236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.144677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.145723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.147623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.147920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.148204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.148482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.148622 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.148917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.149205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.149482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.156998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.157292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.157578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.158061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.158355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.158637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.164564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81763) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.164857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.165141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.165511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.165667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81762) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 25 04:52:00.165771 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.165924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.166011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.166217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.166301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.166502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.166586 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.166781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.166863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.167057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.167147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.167332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.167414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.167596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.167678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.167863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.167945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.168123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81763) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.168204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.168437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.168702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.168972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.169236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81762) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 25 04:52:00.202938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x562b0ce2e9c0 pid: 81763 (default app) Aug 25 04:52:00.204135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x562b0ce2e9c0 pid: 81762 (default app) Aug 25 04:52:00.206355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b45e34d-1fdd-4245-aff5-9afe38b81b5f None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:00.206642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b45e34d-1fdd-4245-aff5-9afe38b81b5f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:00.206912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b45e34d-1fdd-4245-aff5-9afe38b81b5f None None] PATH_INFO: `/v3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:00.211097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1/1] 10.222.0.16 () {50 vars in 791 bytes} [Thu Aug 25 04:52:00 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:07.953812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d04a88-8aab-45d0-8de3-e5eaae4be17f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:07.954384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d04a88-8aab-45d0-8de3-e5eaae4be17f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:07.954384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d04a88-8aab-45d0-8de3-e5eaae4be17f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:07.958635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1/2] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:07 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:52:07.964548 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:07.964957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:07.965086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:08.034445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: INFO dbcounter [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] Registered counter for database keystone Aug 25 04:52:08.055467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG oslo_db.sqlalchemy.engines [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81763) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 25 04:52:08.062035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writer thread running {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 25 04:52:08.238461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG passlib.handlers.bcrypt [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81763) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 25 04:52:08.238885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG passlib.handlers.bcrypt [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81763) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 25 04:52:08.292851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:08.294467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:08.422320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 2/3] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:07 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 459 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Aug 25 04:52:08.876959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d04a88-8aab-45d0-8de3-e5eaae4be17f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:08.944928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: INFO dbcounter [None req-41d04a88-8aab-45d0-8de3-e5eaae4be17f None None] Registered counter for database keystone Aug 25 04:52:08.964250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG oslo_db.sqlalchemy.engines [None req-41d04a88-8aab-45d0-8de3-e5eaae4be17f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81762) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 25 04:52:08.970538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writer thread running {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 25 04:52:09.089252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d04a88-8aab-45d0-8de3-e5eaae4be17f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:09.093563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f21a3bd-9687-42b6-b5d3-e8ea779466d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:09.094223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f21a3bd-9687-42b6-b5d3-e8ea779466d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:09.094501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f21a3bd-9687-42b6-b5d3-e8ea779466d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:09.094768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f21a3bd-9687-42b6-b5d3-e8ea779466d6 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:09.102681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f21a3bd-9687-42b6-b5d3-e8ea779466d6 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:09.155636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING keystone.common.rbac_enforcer.enforcer [None req-0f21a3bd-9687-42b6-b5d3-e8ea779466d6 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 25 04:52:09.155928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f21a3bd-9687-42b6-b5d3-e8ea779466d6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:09.165134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 2/4] 10.222.0.16 () {58 vars in 1324 bytes} [Thu Aug 25 04:52:08 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 25 04:52:09.171756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:09.186003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c03e1ae-4395-4803-8680-13954c9da794 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:09.190417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1426eccd-e6db-4dae-a994-1e279c2aaaff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:09.191509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426eccd-e6db-4dae-a994-1e279c2aaaff admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:09.191792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426eccd-e6db-4dae-a994-1e279c2aaaff admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:09.192066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426eccd-e6db-4dae-a994-1e279c2aaaff admin admin] PATH_INFO: `/v3/projects/f65e42175ca249f08b4d6dc3b72146f4` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:09.193612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1426eccd-e6db-4dae-a994-1e279c2aaaff admin admin] RBAC: Authorizing `identity:get_project(project_id=f65e42175ca249f08b4d6dc3b72146f4)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:09.247329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WARNING keystone.common.rbac_enforcer.enforcer [None req-1426eccd-e6db-4dae-a994-1e279c2aaaff admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 25 04:52:09.247698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1426eccd-e6db-4dae-a994-1e279c2aaaff admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:09.252247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 3/5] 10.222.0.16 () {56 vars in 1207 bytes} [Thu Aug 25 04:52:09 2022] GET /identity/v3/projects/f65e42175ca249f08b4d6dc3b72146f4 => generated 344 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:11.107804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cef9ef-1775-444a-b821-a8a72fa551af None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:11.108337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cef9ef-1775-444a-b821-a8a72fa551af None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:11.108519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cef9ef-1775-444a-b821-a8a72fa551af None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:11.112459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 3/6] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:11 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:11.117755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193a37a7-9324-43ee-ad0c-71f213255a95 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:11.118176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193a37a7-9324-43ee-ad0c-71f213255a95 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:11.118310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193a37a7-9324-43ee-ad0c-71f213255a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:11.138368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-193a37a7-9324-43ee-ad0c-71f213255a95 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:11.139859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-193a37a7-9324-43ee-ad0c-71f213255a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:11.147477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 4/7] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:11 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 25 04:52:11.414371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cef9ef-1775-444a-b821-a8a72fa551af None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:11.426444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cef9ef-1775-444a-b821-a8a72fa551af None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:11.430799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6928644-4134-4419-bb43-d4fb1012bea6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:11.431397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6928644-4134-4419-bb43-d4fb1012bea6 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:11.431676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6928644-4134-4419-bb43-d4fb1012bea6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:11.431948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6928644-4134-4419-bb43-d4fb1012bea6 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:11.440104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6928644-4134-4419-bb43-d4fb1012bea6 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:11.442003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6928644-4134-4419-bb43-d4fb1012bea6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:11.450608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 4/8] 10.222.0.16 () {58 vars in 1324 bytes} [Thu Aug 25 04:52:11 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 04:52:11.455777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193a37a7-9324-43ee-ad0c-71f213255a95 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:11.464955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193a37a7-9324-43ee-ad0c-71f213255a95 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:11.469779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db02bdee-22b5-4df3-a2d1-0fa0a6f8b230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:11.470402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db02bdee-22b5-4df3-a2d1-0fa0a6f8b230 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:11.470680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db02bdee-22b5-4df3-a2d1-0fa0a6f8b230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:11.470947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db02bdee-22b5-4df3-a2d1-0fa0a6f8b230 admin admin] PATH_INFO: `/v3/users/c2010f49b97347fb8aff29794f27d93f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:11.472240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db02bdee-22b5-4df3-a2d1-0fa0a6f8b230 admin admin] RBAC: Authorizing `identity:get_user(user_id=c2010f49b97347fb8aff29794f27d93f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:11.474089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db02bdee-22b5-4df3-a2d1-0fa0a6f8b230 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:11.475631 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 5/9] 10.222.0.16 () {56 vars in 1198 bytes} [Thu Aug 25 04:52:11 2022] GET /identity/v3/users/c2010f49b97347fb8aff29794f27d93f => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:13.401172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afd7f1d-1f54-43a3-8fec-15ed686d04c8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:13.401727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afd7f1d-1f54-43a3-8fec-15ed686d04c8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:13.401832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afd7f1d-1f54-43a3-8fec-15ed686d04c8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:13.403080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 5/10] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:13.408356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688823dd-e00f-4213-aef5-d659f45a8358 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:13.408759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688823dd-e00f-4213-aef5-d659f45a8358 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:13.409073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688823dd-e00f-4213-aef5-d659f45a8358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:13.427462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-688823dd-e00f-4213-aef5-d659f45a8358 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:13.428665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-688823dd-e00f-4213-aef5-d659f45a8358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:13.459102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70b346-2f1c-471b-b1be-13b6f71cf2b0 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:13.459385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70b346-2f1c-471b-b1be-13b6f71cf2b0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:13.459658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70b346-2f1c-471b-b1be-13b6f71cf2b0 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:13.460961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 6/11] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:52:13.466072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:13.466912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:13.467192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:13.507785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG passlib.handlers.bcrypt [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81762) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 25 04:52:13.508136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG passlib.handlers.bcrypt [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81762) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 25 04:52:13.510350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 6/12] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:13.554042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:13.555606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:13.651988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 7/13] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:13.774913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688823dd-e00f-4213-aef5-d659f45a8358 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:13.798655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:13.878088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688823dd-e00f-4213-aef5-d659f45a8358 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:13.910813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426b4845-d974-452a-8d00-e265e42ee7a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:13.911454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b4845-d974-452a-8d00-e265e42ee7a3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:13.911738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b4845-d974-452a-8d00-e265e42ee7a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:13.912015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b4845-d974-452a-8d00-e265e42ee7a3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:13.923029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426b4845-d974-452a-8d00-e265e42ee7a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:13.926386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-426b4845-d974-452a-8d00-e265e42ee7a3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:13.932401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:13.933456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:13.938632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 7/14] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:52:13.939647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426b4845-d974-452a-8d00-e265e42ee7a3 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:14.020646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:14.034925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3918b3-4ae5-43da-bc5b-bb709be3f767 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:14.034925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:14.038604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426b4845-d974-452a-8d00-e265e42ee7a3 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:14.066132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f13817-7251-4b76-bb0d-fcaf2e417c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:14.066979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f13817-7251-4b76-bb0d-fcaf2e417c40 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:14.067261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f13817-7251-4b76-bb0d-fcaf2e417c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:14.067535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f13817-7251-4b76-bb0d-fcaf2e417c40 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:14.072352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:14.073009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:14.073285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:14.073557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:14.074252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:14.074555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:14.075974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:14.087950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 8/15] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:14.093424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:14.133950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f13817-7251-4b76-bb0d-fcaf2e417c40 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:14.135467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f13817-7251-4b76-bb0d-fcaf2e417c40 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:14.190829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abc5ac3-8654-4c86-8cff-ca3f73bf01fd None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:14.231006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c3de1f-ef1e-41e8-9130-36a64251f469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:14.231670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c3de1f-ef1e-41e8-9130-36a64251f469 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:14.231931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c3de1f-ef1e-41e8-9130-36a64251f469 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:14.232200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c3de1f-ef1e-41e8-9130-36a64251f469 None admin] PATH_INFO: `/v3/users/c2010f49b97347fb8aff29794f27d93f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:14.232396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 8/16] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 25 04:52:14.233358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c3de1f-ef1e-41e8-9130-36a64251f469 None admin] RBAC: Authorizing `identity:get_user(user_id=c2010f49b97347fb8aff29794f27d93f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:14.234982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c3de1f-ef1e-41e8-9130-36a64251f469 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:14.236549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 9/17] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:14 2022] GET /identity/v3/users/c2010f49b97347fb8aff29794f27d93f => generated 247 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:14.238855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f13817-7251-4b76-bb0d-fcaf2e417c40 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:14.241641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:14.241878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c3de1f-ef1e-41e8-9130-36a64251f469 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:14.255776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:14.335622 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f13817-7251-4b76-bb0d-fcaf2e417c40 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:14.340668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c3de1f-ef1e-41e8-9130-36a64251f469 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:14.368981 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:14.369868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:14.370152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:14.370426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:14.374468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4f9aec-4ecd-4655-be6f-b8698fc24293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:14.375097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4f9aec-4ecd-4655-be6f-b8698fc24293 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:14.375381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4f9aec-4ecd-4655-be6f-b8698fc24293 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:14.375658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4f9aec-4ecd-4655-be6f-b8698fc24293 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:14.376774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4f9aec-4ecd-4655-be6f-b8698fc24293 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:14.378280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4f9aec-4ecd-4655-be6f-b8698fc24293 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:14.379830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 10/18] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:14.385618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4f9aec-4ecd-4655-be6f-b8698fc24293 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:14.389961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:14.391429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:52:14.410263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:14.411563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:52:14.412438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:52:14.412438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:52:14.412438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:14.412438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 9/19] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:14.417379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:14.467401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:14.472946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:14.486595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4f9aec-4ecd-4655-be6f-b8698fc24293 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:14.516274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb2cfcf-7b19-4e7b-8235-4d17ed9ecbc9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:14.520398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af953b0b-402d-4413-9201-b470745dc9c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:14.521009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af953b0b-402d-4413-9201-b470745dc9c2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:14.521296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af953b0b-402d-4413-9201-b470745dc9c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:14.521616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af953b0b-402d-4413-9201-b470745dc9c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:14.522350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af953b0b-402d-4413-9201-b470745dc9c2 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.domain.id=default, user.id=c2010f49b97347fb8aff29794f27d93f) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:14.522660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af953b0b-402d-4413-9201-b470745dc9c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:14.524112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af953b0b-402d-4413-9201-b470745dc9c2 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:14.531166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 11/20] 10.222.0.16 () {56 vars in 1322 bytes} [Thu Aug 25 04:52:14 2022] GET /identity/v3/role_assignments?user.id=c2010f49b97347fb8aff29794f27d93f&scope.domain.id=default&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 233 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:14.551415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa0a71b-afc4-40a3-b5d7-1d6fc73bab85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:14.552046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa0a71b-afc4-40a3-b5d7-1d6fc73bab85 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:14.552322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa0a71b-afc4-40a3-b5d7-1d6fc73bab85 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:14.552593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa0a71b-afc4-40a3-b5d7-1d6fc73bab85 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:14.553254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa0a71b-afc4-40a3-b5d7-1d6fc73bab85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:14.553554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa0a71b-afc4-40a3-b5d7-1d6fc73bab85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:14.555014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa0a71b-afc4-40a3-b5d7-1d6fc73bab85 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:14.574102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 10/21] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:52:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:16.541570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db3323-6978-4437-9bcc-66f238b2d5c0 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:16.542097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db3323-6978-4437-9bcc-66f238b2d5c0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:16.542097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23db3323-6978-4437-9bcc-66f238b2d5c0 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:16.543604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 12/22] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:16.549091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a9d414-5370-41ed-9153-36bd3da273d4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:16.549488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a9d414-5370-41ed-9153-36bd3da273d4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:16.549691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a9d414-5370-41ed-9153-36bd3da273d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:16.573498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-06a9d414-5370-41ed-9153-36bd3da273d4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:16.574742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-06a9d414-5370-41ed-9153-36bd3da273d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:16.668073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 11/23] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:16.745078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea75f2d5-3c12-4032-8da6-569820a2ba43 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:16.745409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea75f2d5-3c12-4032-8da6-569820a2ba43 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:16.745651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea75f2d5-3c12-4032-8da6-569820a2ba43 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:16.747765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 13/24] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:16.753132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c03a1b-525b-4d0d-9176-0aca8b95c2fb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:16.753416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c03a1b-525b-4d0d-9176-0aca8b95c2fb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:16.753714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c03a1b-525b-4d0d-9176-0aca8b95c2fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:16.782631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d7c03a1b-525b-4d0d-9176-0aca8b95c2fb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:16.783835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d7c03a1b-525b-4d0d-9176-0aca8b95c2fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:16.834780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:16.868194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 12/25] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:17.274231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1454926-525e-432d-a258-3254353433bb None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:17.274511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1454926-525e-432d-a258-3254353433bb None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:17.275012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1454926-525e-432d-a258-3254353433bb None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:17.276792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 14/26] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:17.278004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c03a1b-525b-4d0d-9176-0aca8b95c2fb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:17.297074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0e7de9-478d-4656-b4a3-ba3893336be8 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:17.297710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0e7de9-478d-4656-b4a3-ba3893336be8 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:17.298199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0e7de9-478d-4656-b4a3-ba3893336be8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:17.346665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-cc0e7de9-478d-4656-b4a3-ba3893336be8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:17.348387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cc0e7de9-478d-4656-b4a3-ba3893336be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:17.427156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c03a1b-525b-4d0d-9176-0aca8b95c2fb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:17.435670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 15/27] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Aug 25 04:52:17.436356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0e7de9-478d-4656-b4a3-ba3893336be8 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:17.487928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4e11e0-9f9f-413a-93b1-c8f185c12671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:17.488509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e11e0-9f9f-413a-93b1-c8f185c12671 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:17.488789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e11e0-9f9f-413a-93b1-c8f185c12671 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:17.489060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e11e0-9f9f-413a-93b1-c8f185c12671 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:17.513921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4e11e0-9f9f-413a-93b1-c8f185c12671 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-da4e11e0-9f9f-413a-93b1-c8f185c12671 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:17.520422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:17.521281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:17.522055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:17.526072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 13/28] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:17.527344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c6ce2f-53b4-4263-9334-2bf7f537be1d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:17.527633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c6ce2f-53b4-4263-9334-2bf7f537be1d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:17.527907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c6ce2f-53b4-4263-9334-2bf7f537be1d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:17.529690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 14/29] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:17.533118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c6ce2f-53b4-4263-9334-2bf7f537be1d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:17.583897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:17.589924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:17.606200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0e7de9-478d-4656-b4a3-ba3893336be8 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:17.672310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab80c1d5-bad6-489b-a66b-bea5bdfb2419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:17.672953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80c1d5-bad6-489b-a66b-bea5bdfb2419 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:17.673232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80c1d5-bad6-489b-a66b-bea5bdfb2419 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:17.673502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab80c1d5-bad6-489b-a66b-bea5bdfb2419 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:17.689382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab80c1d5-bad6-489b-a66b-bea5bdfb2419 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:17.690890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab80c1d5-bad6-489b-a66b-bea5bdfb2419 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ab80c1d5-bad6-489b-a66b-bea5bdfb2419 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:52:17.718092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:17.719010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:52:17.719829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:52:17.719829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:52:17.719829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:17.719829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 16/30] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:17.720553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5121123d-1adb-4319-8dea-6c15ac053e76 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:17.720674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5121123d-1adb-4319-8dea-6c15ac053e76 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:17.722224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5121123d-1adb-4319-8dea-6c15ac053e76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:17.725536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c6ce2f-53b4-4263-9334-2bf7f537be1d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:17.755184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5121123d-1adb-4319-8dea-6c15ac053e76 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:17.756292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5121123d-1adb-4319-8dea-6c15ac053e76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:17.781329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:17.782041 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:17.782349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:17.782692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:17.783371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:17.783676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:17.785307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:17.804378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 15/31] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:17.805461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:17.839810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 17/32] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:17.840570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5121123d-1adb-4319-8dea-6c15ac053e76 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:17.903524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:17.953525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5121123d-1adb-4319-8dea-6c15ac053e76 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:17.985023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:17.991048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f934ec-b74a-4dcd-bb26-070c8e1ec4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:17.991298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cbd66e-02a1-4c6e-a612-9c24a67ce833 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:17.991672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f934ec-b74a-4dcd-bb26-070c8e1ec4d1 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:17.991951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f934ec-b74a-4dcd-bb26-070c8e1ec4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:17.992224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f934ec-b74a-4dcd-bb26-070c8e1ec4d1 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:17.993049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f934ec-b74a-4dcd-bb26-070c8e1ec4d1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:17.994482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f934ec-b74a-4dcd-bb26-070c8e1ec4d1 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:18.040408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:18.040408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.040408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.040408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.040882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:18.040882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:18.040882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:18.049820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 16/33] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:52:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:18.050361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:18.170931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e7b552-dc8b-4b0f-97d6-e6ecc77a3fc5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:18.208501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b68e1a5-ab15-48f2-93bb-d8a7613b3949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:18.208501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b68e1a5-ab15-48f2-93bb-d8a7613b3949 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.208501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b68e1a5-ab15-48f2-93bb-d8a7613b3949 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.208501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b68e1a5-ab15-48f2-93bb-d8a7613b3949 None admin] PATH_INFO: `/v3/users/c2010f49b97347fb8aff29794f27d93f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.208945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b68e1a5-ab15-48f2-93bb-d8a7613b3949 None admin] RBAC: Authorizing `identity:get_user(user_id=c2010f49b97347fb8aff29794f27d93f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:18.209280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b68e1a5-ab15-48f2-93bb-d8a7613b3949 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:18.211089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 17/34] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:18 2022] GET /identity/v3/users/c2010f49b97347fb8aff29794f27d93f => generated 247 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:18.211607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b68e1a5-ab15-48f2-93bb-d8a7613b3949 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:18.240648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:18.269849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 18/35] 10.222.0.16 () {60 vars in 1131 bytes} [Thu Aug 25 04:52:17 2022] POST /identity/v3/roles => generated 219 bytes in 426 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:18.269849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bfc1e-4009-407b-9d8d-dc9bf2224525 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.269849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bfc1e-4009-407b-9d8d-dc9bf2224525 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.269849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bfc1e-4009-407b-9d8d-dc9bf2224525 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.269849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 19/36] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:18.274689 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53bfc1e-4009-407b-9d8d-dc9bf2224525 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:18.330929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b68e1a5-ab15-48f2-93bb-d8a7613b3949 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:18.370600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246fe46b-2464-40f9-927d-92ab9487cf3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:18.371236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246fe46b-2464-40f9-927d-92ab9487cf3b None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.371517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246fe46b-2464-40f9-927d-92ab9487cf3b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.371799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246fe46b-2464-40f9-927d-92ab9487cf3b None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.373183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246fe46b-2464-40f9-927d-92ab9487cf3b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:18.375061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246fe46b-2464-40f9-927d-92ab9487cf3b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:18.400987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53bfc1e-4009-407b-9d8d-dc9bf2224525 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:18.420505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:18.439606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3290b088-b715-48ed-9afd-165430ebbf8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:18.440224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3290b088-b715-48ed-9afd-165430ebbf8a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.440508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3290b088-b715-48ed-9afd-165430ebbf8a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.444925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3290b088-b715-48ed-9afd-165430ebbf8a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.444925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3290b088-b715-48ed-9afd-165430ebbf8a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:18.444925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3290b088-b715-48ed-9afd-165430ebbf8a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:18.445285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 20/37] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:18.445969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3290b088-b715-48ed-9afd-165430ebbf8a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:18.548874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3290b088-b715-48ed-9afd-165430ebbf8a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:18.581341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c3cfdb-3ddb-497a-ac66-a04f1621ea26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:18.581919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3cfdb-3ddb-497a-ac66-a04f1621ea26 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.582200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3cfdb-3ddb-497a-ac66-a04f1621ea26 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.582471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3cfdb-3ddb-497a-ac66-a04f1621ea26 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.583122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c3cfdb-3ddb-497a-ac66-a04f1621ea26 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:18.584499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c3cfdb-3ddb-497a-ac66-a04f1621ea26 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:18.603036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 18/38] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:52:18 2022] POST /identity/v3/projects => generated 301 bytes in 392 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:18.604464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb8bc9-8c8b-4f54-a1c5-7373b27508bd None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.604754 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb8bc9-8c8b-4f54-a1c5-7373b27508bd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.605032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb8bc9-8c8b-4f54-a1c5-7373b27508bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.626390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-30bb8bc9-8c8b-4f54-a1c5-7373b27508bd None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:18.628686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-30bb8bc9-8c8b-4f54-a1c5-7373b27508bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:18.654761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 21/39] 10.222.0.16 () {60 vars in 1131 bytes} [Thu Aug 25 04:52:18 2022] POST /identity/v3/roles => generated 225 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:18.662292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50749aa-8fac-4ce9-91f7-7fa93654315c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.662292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50749aa-8fac-4ce9-91f7-7fa93654315c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.662292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50749aa-8fac-4ce9-91f7-7fa93654315c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.662292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 22/40] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:18.662292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50749aa-8fac-4ce9-91f7-7fa93654315c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:18.727558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 19/41] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:18.729147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd77fb6-55a2-486d-9632-a1bec6399bf5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.729446 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd77fb6-55a2-486d-9632-a1bec6399bf5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.729780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd77fb6-55a2-486d-9632-a1bec6399bf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.752465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6cd77fb6-55a2-486d-9632-a1bec6399bf5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:18.754132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6cd77fb6-55a2-486d-9632-a1bec6399bf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:18.757949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:18.850015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 20/42] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:18.855413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:18.913444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50749aa-8fac-4ce9-91f7-7fa93654315c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:18.951491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0e0f91-c4e7-485e-98e0-1c95febed307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:18.952223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0e0f91-c4e7-485e-98e0-1c95febed307 None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:18.952515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0e0f91-c4e7-485e-98e0-1c95febed307 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:18.952798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0e0f91-c4e7-485e-98e0-1c95febed307 None admin] PATH_INFO: `/v3/domains/default/users/c2010f49b97347fb8aff29794f27d93f/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:18.954914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0e0f91-c4e7-485e-98e0-1c95febed307 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=c2010f49b97347fb8aff29794f27d93f, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:18.956383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0e0f91-c4e7-485e-98e0-1c95febed307 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:19.023035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd77fb6-55a2-486d-9632-a1bec6399bf5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:19.051043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 23/43] 10.222.0.16 () {58 vars in 1361 bytes} [Thu Aug 25 04:52:18 2022] PUT /identity/v3/domains/default/users/c2010f49b97347fb8aff29794f27d93f/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:19.139540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd77fb6-55a2-486d-9632-a1bec6399bf5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:19.149965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0e0f91-c4e7-485e-98e0-1c95febed307 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:19.177932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0449337-9380-413b-9fc5-b1969b3731ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:19.178559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0449337-9380-413b-9fc5-b1969b3731ad None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:19.178839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0449337-9380-413b-9fc5-b1969b3731ad None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:19.179113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0449337-9380-413b-9fc5-b1969b3731ad None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:19.179847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0449337-9380-413b-9fc5-b1969b3731ad None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:19.181341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0449337-9380-413b-9fc5-b1969b3731ad None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:19.257017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0e0f91-c4e7-485e-98e0-1c95febed307 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:19.259326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:19.269910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:19.276051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 21/44] 10.222.0.16 () {60 vars in 1131 bytes} [Thu Aug 25 04:52:19 2022] POST /identity/v3/roles => generated 223 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:52:19.293420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048ad6f4-e5de-4738-8246-11e36d540ae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:19.294193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048ad6f4-e5de-4738-8246-11e36d540ae0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:19.294480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048ad6f4-e5de-4738-8246-11e36d540ae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:19.294755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048ad6f4-e5de-4738-8246-11e36d540ae0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:19.295995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048ad6f4-e5de-4738-8246-11e36d540ae0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:19.297621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048ad6f4-e5de-4738-8246-11e36d540ae0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:19.299551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 24/45] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:19.305228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0449337-9380-413b-9fc5-b1969b3731ad None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:19.416886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0449337-9380-413b-9fc5-b1969b3731ad None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:19.455193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3340bd2d-e623-40d4-bb58-75df8b79dd5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:19.455815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340bd2d-e623-40d4-bb58-75df8b79dd5b None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:19.456100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340bd2d-e623-40d4-bb58-75df8b79dd5b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:19.456379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340bd2d-e623-40d4-bb58-75df8b79dd5b None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:19.457039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3340bd2d-e623-40d4-bb58-75df8b79dd5b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:19.458547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3340bd2d-e623-40d4-bb58-75df8b79dd5b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:19.575872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 22/46] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:52:19 2022] POST /identity/v3/projects => generated 312 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:20.978934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32016d4b-25a3-48ee-9356-21246cf44b24 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:20.979470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32016d4b-25a3-48ee-9356-21246cf44b24 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:20.979470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32016d4b-25a3-48ee-9356-21246cf44b24 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:20.980826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 25/47] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:20.986640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce99c17-55cd-48af-bbe4-ba896a60641e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:20.987027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce99c17-55cd-48af-bbe4-ba896a60641e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:20.987325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce99c17-55cd-48af-bbe4-ba896a60641e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:21.006635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9ce99c17-55cd-48af-bbe4-ba896a60641e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:21.007824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9ce99c17-55cd-48af-bbe4-ba896a60641e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:21.156521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:21.176720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 23/48] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:21.444611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32016d4b-25a3-48ee-9356-21246cf44b24 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:21.480462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfacb0-9d8f-4d72-97a1-68ff2b8a6489 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:21.480763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfacb0-9d8f-4d72-97a1-68ff2b8a6489 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:21.481865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bfacb0-9d8f-4d72-97a1-68ff2b8a6489 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:21.483225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 24/49] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:52:21.488185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29868e3-4078-4a91-ac06-4964b47588ba None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:21.488470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29868e3-4078-4a91-ac06-4964b47588ba None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:21.488745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29868e3-4078-4a91-ac06-4964b47588ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:21.506434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a29868e3-4078-4a91-ac06-4964b47588ba None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:21.507614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a29868e3-4078-4a91-ac06-4964b47588ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:21.546934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32016d4b-25a3-48ee-9356-21246cf44b24 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:21.573755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 25/50] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:21.580126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af9c48e-3160-4251-985a-365e23383b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:21.580790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9c48e-3160-4251-985a-365e23383b0a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:21.581071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9c48e-3160-4251-985a-365e23383b0a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:21.581343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9c48e-3160-4251-985a-365e23383b0a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:21.591953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af9c48e-3160-4251-985a-365e23383b0a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-0af9c48e-3160-4251-985a-365e23383b0a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:21.596750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:21.597558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:21.598349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 26/51] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:21.603454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29868e3-4078-4a91-ac06-4964b47588ba None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:21.698803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29868e3-4078-4a91-ac06-4964b47588ba None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:21.709384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:21.732189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:21.732785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:21.733066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:21.733340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:21.734081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:21.734381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:21.735748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:21.743765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 26/52] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:21.749240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af9c48e-3160-4251-985a-365e23383b0a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:21.781860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:21.844129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:21.850266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af9c48e-3160-4251-985a-365e23383b0a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:21.884984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4226c14e-011a-4fbb-920d-9234a73f5c31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:21.885652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4226c14e-011a-4fbb-920d-9234a73f5c31 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:21.885945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4226c14e-011a-4fbb-920d-9234a73f5c31 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:21.886209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4226c14e-011a-4fbb-920d-9234a73f5c31 None admin] PATH_INFO: `/v3/users/c2010f49b97347fb8aff29794f27d93f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:21.887318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4226c14e-011a-4fbb-920d-9234a73f5c31 None admin] RBAC: Authorizing `identity:get_user(user_id=c2010f49b97347fb8aff29794f27d93f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:21.889358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4226c14e-011a-4fbb-920d-9234a73f5c31 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:21.890955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 27/53] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:21 2022] GET /identity/v3/users/c2010f49b97347fb8aff29794f27d93f => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:21.896357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4226c14e-011a-4fbb-920d-9234a73f5c31 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:21.945196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554da923-2a1c-40a5-b383-175a23555e46 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:21.978628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e42057-c18a-4ca6-ae9b-a10a86a1d9de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:21.979191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e42057-c18a-4ca6-ae9b-a10a86a1d9de None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:21.979469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e42057-c18a-4ca6-ae9b-a10a86a1d9de None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:21.979740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e42057-c18a-4ca6-ae9b-a10a86a1d9de None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:21.980840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e42057-c18a-4ca6-ae9b-a10a86a1d9de None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:21.982342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e42057-c18a-4ca6-ae9b-a10a86a1d9de None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:21.983929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 27/54] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:21.990422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e42057-c18a-4ca6-ae9b-a10a86a1d9de None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:21.990989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:21.994193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4226c14e-011a-4fbb-920d-9234a73f5c31 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:22.027972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39d2ad-2fbd-48f6-ac4c-b87a804dc921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:22.029147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39d2ad-2fbd-48f6-ac4c-b87a804dc921 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:22.029425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39d2ad-2fbd-48f6-ac4c-b87a804dc921 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:22.029734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39d2ad-2fbd-48f6-ac4c-b87a804dc921 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:22.030852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b39d2ad-2fbd-48f6-ac4c-b87a804dc921 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:22.032306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b39d2ad-2fbd-48f6-ac4c-b87a804dc921 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:22.033892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 28/55] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:22.035525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:22.039507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39d2ad-2fbd-48f6-ac4c-b87a804dc921 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:22.093983 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e42057-c18a-4ca6-ae9b-a10a86a1d9de None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:22.128128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e80640a-77fc-4823-abb1-58cc3772bb4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:22.128736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e80640a-77fc-4823-abb1-58cc3772bb4e None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:22.129018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e80640a-77fc-4823-abb1-58cc3772bb4e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:22.129293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e80640a-77fc-4823-abb1-58cc3772bb4e None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:22.129965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e80640a-77fc-4823-abb1-58cc3772bb4e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:22.131443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e80640a-77fc-4823-abb1-58cc3772bb4e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:22.136764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39d2ad-2fbd-48f6-ac4c-b87a804dc921 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:22.170733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57d6a8e-4afd-46d8-abdb-de6736420ac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:22.171333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57d6a8e-4afd-46d8-abdb-de6736420ac1 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:22.171616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57d6a8e-4afd-46d8-abdb-de6736420ac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:22.171890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57d6a8e-4afd-46d8-abdb-de6736420ac1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:22.172606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57d6a8e-4afd-46d8-abdb-de6736420ac1 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.domain.id=default, user.id=c2010f49b97347fb8aff29794f27d93f) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:22.172915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57d6a8e-4afd-46d8-abdb-de6736420ac1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:22.174373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57d6a8e-4afd-46d8-abdb-de6736420ac1 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:22.179965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 29/56] 10.222.0.16 () {56 vars in 1322 bytes} [Thu Aug 25 04:52:22 2022] GET /identity/v3/role_assignments?user.id=c2010f49b97347fb8aff29794f27d93f&scope.domain.id=default&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 530 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:22.233842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 28/57] 10.222.0.16 () {60 vars in 1140 bytes} [Thu Aug 25 04:52:21 2022] POST /identity/v3/projects => generated 298 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:52:24.119011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f59d7-80e5-4567-8cdc-67dfa0fc25c4 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:24.119556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f59d7-80e5-4567-8cdc-67dfa0fc25c4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:24.119556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f59d7-80e5-4567-8cdc-67dfa0fc25c4 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:24.120937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 30/58] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:24.126811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceed1cc-4513-455f-93d1-76c689373898 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:24.127173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceed1cc-4513-455f-93d1-76c689373898 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:24.127376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ceed1cc-4513-455f-93d1-76c689373898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:24.145028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8ceed1cc-4513-455f-93d1-76c689373898 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:24.146250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8ceed1cc-4513-455f-93d1-76c689373898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:24.210498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 29/59] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:24.480644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9f59d7-80e5-4567-8cdc-67dfa0fc25c4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:24.512167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:24.580927 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9f59d7-80e5-4567-8cdc-67dfa0fc25c4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:24.614006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c11a491-4ffa-481c-ad81-899e938233df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:24.614624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c11a491-4ffa-481c-ad81-899e938233df None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:24.614905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c11a491-4ffa-481c-ad81-899e938233df None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:24.615178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c11a491-4ffa-481c-ad81-899e938233df None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:24.616271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c11a491-4ffa-481c-ad81-899e938233df None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:24.617839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c11a491-4ffa-481c-ad81-899e938233df None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:24.619381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 31/60] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:24.625527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceed1cc-4513-455f-93d1-76c689373898 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:24.649570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:24.723176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ceed1cc-4513-455f-93d1-76c689373898 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:24.755949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b491ee-5b06-49b0-b360-f037298615c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:24.756494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b491ee-5b06-49b0-b360-f037298615c4 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:24.756774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b491ee-5b06-49b0-b360-f037298615c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:24.757045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b491ee-5b06-49b0-b360-f037298615c4 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:24.757706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b491ee-5b06-49b0-b360-f037298615c4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:24.759135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b491ee-5b06-49b0-b360-f037298615c4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:24.856816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 30/61] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:52:24 2022] POST /identity/v3/users => generated 275 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:27.866451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94335c7a-d869-4a3f-9540-cd72a8a2bbc2 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:27.867445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94335c7a-d869-4a3f-9540-cd72a8a2bbc2 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:27.867924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94335c7a-d869-4a3f-9540-cd72a8a2bbc2 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:27.869736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 32/62] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:27.875309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc0213a-bfe4-4915-99f2-353cd43d1c23 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:27.876149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc0213a-bfe4-4915-99f2-353cd43d1c23 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:27.876617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc0213a-bfe4-4915-99f2-353cd43d1c23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:27.902754 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4cc0213a-bfe4-4915-99f2-353cd43d1c23 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:27.904456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4cc0213a-bfe4-4915-99f2-353cd43d1c23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:27.905159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3fd14-80b6-425f-b59a-66d805f958bd None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:27.905703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3fd14-80b6-425f-b59a-66d805f958bd None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:27.906237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf3fd14-80b6-425f-b59a-66d805f958bd None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:27.907876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 33/63] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:52:27.913120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f15f7-326e-4eb3-9019-f66e23f2ec9b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:27.913665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f15f7-326e-4eb3-9019-f66e23f2ec9b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:27.914156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f15f7-326e-4eb3-9019-f66e23f2ec9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:27.940415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-b62f15f7-326e-4eb3-9019-f66e23f2ec9b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:27.943311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b62f15f7-326e-4eb3-9019-f66e23f2ec9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:28.035373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 31/64] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:28.055205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0199a77-eba1-433f-b98e-e01b12ffbbd6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.055538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0199a77-eba1-433f-b98e-e01b12ffbbd6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.055764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0199a77-eba1-433f-b98e-e01b12ffbbd6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.057123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 32/65] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:52:28.066937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdefe0f6-5763-43af-a440-090ccb5f6ce4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.067225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdefe0f6-5763-43af-a440-090ccb5f6ce4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.067503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdefe0f6-5763-43af-a440-090ccb5f6ce4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.071062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 34/66] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:28.090021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fdefe0f6-5763-43af-a440-090ccb5f6ce4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:28.092011 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fdefe0f6-5763-43af-a440-090ccb5f6ce4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:28.094152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:28.178524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 33/67] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:28.376295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f15f7-326e-4eb3-9019-f66e23f2ec9b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:28.493893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:28.494033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04400e00-f76e-4c7b-9680-48815e031d6b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.494033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04400e00-f76e-4c7b-9680-48815e031d6b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.494033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04400e00-f76e-4c7b-9680-48815e031d6b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.494033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 34/68] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:28.494033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b0374d-96f4-43d2-8645-5ed2573d7aa3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.494033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b0374d-96f4-43d2-8645-5ed2573d7aa3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.494423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b0374d-96f4-43d2-8645-5ed2573d7aa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.511468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-71b0374d-96f4-43d2-8645-5ed2573d7aa3 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:28.512061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f15f7-326e-4eb3-9019-f66e23f2ec9b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:28.512831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-71b0374d-96f4-43d2-8645-5ed2573d7aa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:28.546429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c7d996-d2c5-4c38-bbc0-f62e4486e083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:28.547068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c7d996-d2c5-4c38-bbc0-f62e4486e083 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.547347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c7d996-d2c5-4c38-bbc0-f62e4486e083 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.547619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c7d996-d2c5-4c38-bbc0-f62e4486e083 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.557421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c7d996-d2c5-4c38-bbc0-f62e4486e083 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-26c7d996-d2c5-4c38-bbc0-f62e4486e083 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:28.562759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:28.563795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 35/69] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:28.564591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c7d996-d2c5-4c38-bbc0-f62e4486e083 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:28.581377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 35/70] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:28.586939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513a497-2251-45f2-abeb-14c36f91c48c None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.587481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513a497-2251-45f2-abeb-14c36f91c48c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.587763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2513a497-2251-45f2-abeb-14c36f91c48c None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.589163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 36/71] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:28.589851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2513a497-2251-45f2-abeb-14c36f91c48c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:28.658859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:28.663620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:28.673747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c7d996-d2c5-4c38-bbc0-f62e4486e083 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:28.692740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2513a497-2251-45f2-abeb-14c36f91c48c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:28.708578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f18fab-7240-4c13-a37a-48ef80c96e90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:28.709213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f18fab-7240-4c13-a37a-48ef80c96e90 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.709496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f18fab-7240-4c13-a37a-48ef80c96e90 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.709796 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f18fab-7240-4c13-a37a-48ef80c96e90 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.720417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f18fab-7240-4c13-a37a-48ef80c96e90 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-c7f18fab-7240-4c13-a37a-48ef80c96e90 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:28.725876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:28.726714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:28.727478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 36/72] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:52:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:28.728766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f18fab-7240-4c13-a37a-48ef80c96e90 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:28.740006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a10fb9-f228-416b-9146-5de3c8641c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:28.741014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a10fb9-f228-416b-9146-5de3c8641c4a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.741301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a10fb9-f228-416b-9146-5de3c8641c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.741923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a10fb9-f228-416b-9146-5de3c8641c4a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.753399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a10fb9-f228-416b-9146-5de3c8641c4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-24a10fb9-f228-416b-9146-5de3c8641c4a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:28.758619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:28.759628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:28.760557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 37/73] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:28.761349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b86eb7-6447-4883-a90d-e3410e5a6787 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.761349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b86eb7-6447-4883-a90d-e3410e5a6787 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.761492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b86eb7-6447-4883-a90d-e3410e5a6787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.784001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-13b86eb7-6447-4883-a90d-e3410e5a6787 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:28.785199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-13b86eb7-6447-4883-a90d-e3410e5a6787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:28.836508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f18fab-7240-4c13-a37a-48ef80c96e90 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:28.860453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 38/74] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:28.861306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b86eb7-6447-4883-a90d-e3410e5a6787 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:28.870788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:28.871401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:28.871728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:28.871984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:28.872643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:28.872944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:28.874350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:28.882986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 37/75] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:28.883820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:28.899518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:28.932569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:28.974854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b86eb7-6447-4883-a90d-e3410e5a6787 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:28.987307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b55f9b-8d8e-46ab-9c94-e3dc6a2287a2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.008858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.009409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.009738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.010013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.010943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:29.011245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.012872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.025932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 39/76] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:52:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:29.025932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.028455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.029077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.029359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.029658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.030329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:29.030647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.032026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.040545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 38/77] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:29.041174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.123893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:29.133427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58648d0e-1e8e-458d-8fdf-7c1764354da4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.148691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48000a73-8f06-479d-ab6f-a75c944403d5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.154273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:29.167780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f17d8b-2f03-43e8-84be-a9501627d0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.168414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f17d8b-2f03-43e8-84be-a9501627d0c8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.168719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f17d8b-2f03-43e8-84be-a9501627d0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.168992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f17d8b-2f03-43e8-84be-a9501627d0c8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.170140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f17d8b-2f03-43e8-84be-a9501627d0c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.171579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f17d8b-2f03-43e8-84be-a9501627d0c8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.173135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 40/78] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:29.173914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f17d8b-2f03-43e8-84be-a9501627d0c8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.183220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f624a355-7f90-4ad3-928f-0d4d57abc880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.183889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f624a355-7f90-4ad3-928f-0d4d57abc880 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.184378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f624a355-7f90-4ad3-928f-0d4d57abc880 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.184493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f624a355-7f90-4ad3-928f-0d4d57abc880 None admin] PATH_INFO: `/v3/users/c2010f49b97347fb8aff29794f27d93f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.185683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f624a355-7f90-4ad3-928f-0d4d57abc880 None admin] RBAC: Authorizing `identity:get_user(user_id=c2010f49b97347fb8aff29794f27d93f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.187073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f624a355-7f90-4ad3-928f-0d4d57abc880 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.188638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 39/79] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/users/c2010f49b97347fb8aff29794f27d93f => generated 247 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:29.189420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f624a355-7f90-4ad3-928f-0d4d57abc880 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.277648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f17d8b-2f03-43e8-84be-a9501627d0c8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.288418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f624a355-7f90-4ad3-928f-0d4d57abc880 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.312049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ed8f0a-43a7-4115-8bb1-0ec9e544e71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.312678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ed8f0a-43a7-4115-8bb1-0ec9e544e71e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.312956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ed8f0a-43a7-4115-8bb1-0ec9e544e71e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.313229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ed8f0a-43a7-4115-8bb1-0ec9e544e71e None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.321897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c063bd-5dd3-4187-a4dd-9acd7804fc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.322527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c063bd-5dd3-4187-a4dd-9acd7804fc4d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.322806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c063bd-5dd3-4187-a4dd-9acd7804fc4d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.323082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c063bd-5dd3-4187-a4dd-9acd7804fc4d None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.328251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ed8f0a-43a7-4115-8bb1-0ec9e544e71e None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.329756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ed8f0a-43a7-4115-8bb1-0ec9e544e71e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.331393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 41/80] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:29.336277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ed8f0a-43a7-4115-8bb1-0ec9e544e71e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.340299 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:29.343672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c063bd-5dd3-4187-a4dd-9acd7804fc4d None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.345460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c063bd-5dd3-4187-a4dd-9acd7804fc4d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.347164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 40/81] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:29.347910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c063bd-5dd3-4187-a4dd-9acd7804fc4d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.379238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:29.430430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ed8f0a-43a7-4115-8bb1-0ec9e544e71e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.451300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c063bd-5dd3-4187-a4dd-9acd7804fc4d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.465040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b587e2a-68a3-414d-82ef-6684d6f8ca4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.466203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b587e2a-68a3-414d-82ef-6684d6f8ca4a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.466480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b587e2a-68a3-414d-82ef-6684d6f8ca4a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.466751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b587e2a-68a3-414d-82ef-6684d6f8ca4a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.476640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b587e2a-68a3-414d-82ef-6684d6f8ca4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1b587e2a-68a3-414d-82ef-6684d6f8ca4a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:29.481987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:29.482842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 42/82] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:29.486178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b587e2a-68a3-414d-82ef-6684d6f8ca4a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.490135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca202335-94e2-4f2e-a86b-5943e4058b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.490781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca202335-94e2-4f2e-a86b-5943e4058b8a None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.491068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca202335-94e2-4f2e-a86b-5943e4058b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.491349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca202335-94e2-4f2e-a86b-5943e4058b8a None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.492001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca202335-94e2-4f2e-a86b-5943e4058b8a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.493506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca202335-94e2-4f2e-a86b-5943e4058b8a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.582662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b587e2a-68a3-414d-82ef-6684d6f8ca4a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.589029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 41/83] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:52:29 2022] POST /identity/v3/users => generated 289 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:29.589878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca202335-94e2-4f2e-a86b-5943e4058b8a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.591298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:29.621652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ccf66-727a-4e5d-b14a-51b33f2932fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.622294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ccf66-727a-4e5d-b14a-51b33f2932fe None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.622576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ccf66-727a-4e5d-b14a-51b33f2932fe None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.622859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ccf66-727a-4e5d-b14a-51b33f2932fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.672379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:29.706408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24ccf66-727a-4e5d-b14a-51b33f2932fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.707909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24ccf66-727a-4e5d-b14a-51b33f2932fe None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.723817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca202335-94e2-4f2e-a86b-5943e4058b8a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.763384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c399bd-4641-416b-adb9-2f8d3a138f92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.764282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c399bd-4641-416b-adb9-2f8d3a138f92 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.764563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c399bd-4641-416b-adb9-2f8d3a138f92 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.764836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c399bd-4641-416b-adb9-2f8d3a138f92 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:29.829565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 43/84] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:29.830762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ccf66-727a-4e5d-b14a-51b33f2932fe None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.840276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:29.848625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c399bd-4641-416b-adb9-2f8d3a138f92 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:29.850531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c399bd-4641-416b-adb9-2f8d3a138f92 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:29.917205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:29.954577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ccf66-727a-4e5d-b14a-51b33f2932fe None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:29.973528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 42/85] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:29.974844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c399bd-4641-416b-adb9-2f8d3a138f92 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:29.991709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ae570a-839c-40b7-af76-8cd8e0f29b2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:29.992324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ae570a-839c-40b7-af76-8cd8e0f29b2a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:29.992610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ae570a-839c-40b7-af76-8cd8e0f29b2a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:29.992887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ae570a-839c-40b7-af76-8cd8e0f29b2a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.063410 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ae570a-839c-40b7-af76-8cd8e0f29b2a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.064987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ae570a-839c-40b7-af76-8cd8e0f29b2a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.073309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:30.085799 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c399bd-4641-416b-adb9-2f8d3a138f92 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.125037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.126118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.126680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.127158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.128019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:30.128648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.130394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.139532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 43/86] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:30.140689 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:30.170388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 44/87] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:30.171134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ae570a-839c-40b7-af76-8cd8e0f29b2a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:30.176621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:30.260371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d3ec67-72f8-498a-ab34-667ea98ca67e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.279207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ae570a-839c-40b7-af76-8cd8e0f29b2a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.287274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:30.299416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e722196b-bfe7-467d-b80f-3f2f07e13c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.300272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722196b-bfe7-467d-b80f-3f2f07e13c63 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.300860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722196b-bfe7-467d-b80f-3f2f07e13c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.300973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722196b-bfe7-467d-b80f-3f2f07e13c63 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.302321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e722196b-bfe7-467d-b80f-3f2f07e13c63 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.303979 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e722196b-bfe7-467d-b80f-3f2f07e13c63 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.305686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 44/88] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:30 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:30.306537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e722196b-bfe7-467d-b80f-3f2f07e13c63 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:30.316273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ab11be-27c5-4f50-b027-143f79593503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.317206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab11be-27c5-4f50-b027-143f79593503 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.317771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab11be-27c5-4f50-b027-143f79593503 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.318282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab11be-27c5-4f50-b027-143f79593503 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.319704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ab11be-27c5-4f50-b027-143f79593503 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.321330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ab11be-27c5-4f50-b027-143f79593503 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.323203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 45/89] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:30 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:30.324243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ab11be-27c5-4f50-b027-143f79593503 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:30.436037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ab11be-27c5-4f50-b027-143f79593503 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.464176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:30.481705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ccdb9a-8ac7-487d-9c00-724ef5a68460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.482641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ccdb9a-8ac7-487d-9c00-724ef5a68460 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.482948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e722196b-bfe7-467d-b80f-3f2f07e13c63 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.483348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ccdb9a-8ac7-487d-9c00-724ef5a68460 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.483829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ccdb9a-8ac7-487d-9c00-724ef5a68460 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.486158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ccdb9a-8ac7-487d-9c00-724ef5a68460 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.487820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ccdb9a-8ac7-487d-9c00-724ef5a68460 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.489683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 46/90] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:30 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:30.491355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ccdb9a-8ac7-487d-9c00-724ef5a68460 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:30.532656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c813d4bb-76ec-4aa2-9c80-11944fc370eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.534058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c813d4bb-76ec-4aa2-9c80-11944fc370eb None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.534774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c813d4bb-76ec-4aa2-9c80-11944fc370eb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.535291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c813d4bb-76ec-4aa2-9c80-11944fc370eb None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.536947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c813d4bb-76ec-4aa2-9c80-11944fc370eb None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.539521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c813d4bb-76ec-4aa2-9c80-11944fc370eb None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.541642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 45/91] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:30 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:30.543219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c813d4bb-76ec-4aa2-9c80-11944fc370eb None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:30.547441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:30.609409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ccdb9a-8ac7-487d-9c00-724ef5a68460 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.645976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.646687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.647045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.647395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.648197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=8419cf9f31bb4753946ad654d60e9e26, user.id=c2010f49b97347fb8aff29794f27d93f) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:30.648580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.650228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.659286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 47/92] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:30 2022] GET /identity/v3/role_assignments?user.id=c2010f49b97347fb8aff29794f27d93f&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:30.660290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:30.685087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c813d4bb-76ec-4aa2-9c80-11944fc370eb None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.727103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.727851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.728205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.728595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.729410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] RBAC: Adding query filter params (role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=8419cf9f31bb4753946ad654d60e9e26, user.id=055c8932e5b14530ad56960260a63d7e) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:30.729819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.732146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.738586 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 46/93] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:30 2022] GET /identity/v3/role_assignments?user.id=055c8932e5b14530ad56960260a63d7e&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:30.739487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:30.762206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:30.807624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bee4e2-dc48-4fa2-af78-5efd411bb582 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.822759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:30.872790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb569b-7b7b-425c-9659-fee9b83e8a65 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:30.919505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f7774b-d0e2-4a18-8dfd-a00a0e884f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.921015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f7774b-d0e2-4a18-8dfd-a00a0e884f61 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.921509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f7774b-d0e2-4a18-8dfd-a00a0e884f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.922006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f7774b-d0e2-4a18-8dfd-a00a0e884f61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.935978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:30.936613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:30.936899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:30.937178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:30.937956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] RBAC: Adding query filter params (role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=8419cf9f31bb4753946ad654d60e9e26, user.id=055c8932e5b14530ad56960260a63d7e) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:30.938272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:30.939746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:30.955498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 48/94] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:30 2022] GET /identity/v3/role_assignments?user.id=055c8932e5b14530ad56960260a63d7e&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 259 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:31.002128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f7774b-d0e2-4a18-8dfd-a00a0e884f61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:31.004125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f7774b-d0e2-4a18-8dfd-a00a0e884f61 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:31.028408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:31.116342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 47/95] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:31.122771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:31.285339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287d441-cfb4-4f53-b37d-68dfc4e35f10 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:31.334414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113ebc5b-9ba5-48d7-8685-220ff6288d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:31.335064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ebc5b-9ba5-48d7-8685-220ff6288d40 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:31.335349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ebc5b-9ba5-48d7-8685-220ff6288d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:31.335628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ebc5b-9ba5-48d7-8685-220ff6288d40 None admin] PATH_INFO: `/v3/projects/5492b71bdd6c44b6b273375d9450b759` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:31.337603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113ebc5b-9ba5-48d7-8685-220ff6288d40 None admin] RBAC: Authorizing `identity:get_project(project_id=5492b71bdd6c44b6b273375d9450b759)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:31.339079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113ebc5b-9ba5-48d7-8685-220ff6288d40 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:31.340803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 49/96] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:31 2022] GET /identity/v3/projects/5492b71bdd6c44b6b273375d9450b759 => generated 312 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:31.346672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f7774b-d0e2-4a18-8dfd-a00a0e884f61 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:31.492773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f7774b-d0e2-4a18-8dfd-a00a0e884f61 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:31.513620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:31.535667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd9fc9e-07de-4de3-9d03-eb286f4fc5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:31.536621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd9fc9e-07de-4de3-9d03-eb286f4fc5b0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:31.537111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd9fc9e-07de-4de3-9d03-eb286f4fc5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:31.537543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd9fc9e-07de-4de3-9d03-eb286f4fc5b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:31.538557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd9fc9e-07de-4de3-9d03-eb286f4fc5b0 None admin] RBAC: Adding query filter params (role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=5492b71bdd6c44b6b273375d9450b759, user.id=055c8932e5b14530ad56960260a63d7e) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:31.539129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd9fc9e-07de-4de3-9d03-eb286f4fc5b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:31.540880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd9fc9e-07de-4de3-9d03-eb286f4fc5b0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:31.547518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 48/97] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:31 2022] GET /identity/v3/role_assignments?user.id=055c8932e5b14530ad56960260a63d7e&scope.project.id=5492b71bdd6c44b6b273375d9450b759&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 259 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:32.257251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba83d19-9f4d-4cb3-a881-e3f8ba68b2b2 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:32.257769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba83d19-9f4d-4cb3-a881-e3f8ba68b2b2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:32.257874 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba83d19-9f4d-4cb3-a881-e3f8ba68b2b2 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:32.259190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 50/98] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:32.265156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc71615-3bf5-4d32-a279-9ebb7d291038 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:32.265550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc71615-3bf5-4d32-a279-9ebb7d291038 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:32.265770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc71615-3bf5-4d32-a279-9ebb7d291038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:32.268985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbb317d-9692-43e7-9930-71843f0fead7 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:32.269270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbb317d-9692-43e7-9930-71843f0fead7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:32.269545 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbb317d-9692-43e7-9930-71843f0fead7 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:32.271155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 51/99] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:32.275654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2adc54-589d-4f19-96cf-10f6a0ac85ee None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:32.275947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2adc54-589d-4f19-96cf-10f6a0ac85ee None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:32.276875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2adc54-589d-4f19-96cf-10f6a0ac85ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:32.318772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8bc71615-3bf5-4d32-a279-9ebb7d291038 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:32.320336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8bc71615-3bf5-4d32-a279-9ebb7d291038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:32.323383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bb2adc54-589d-4f19-96cf-10f6a0ac85ee None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:32.328583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bb2adc54-589d-4f19-96cf-10f6a0ac85ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:32.364813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:32.433106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 52/100] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:32.476270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 49/101] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:32.894349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2adc54-589d-4f19-96cf-10f6a0ac85ee None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:33.025867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2adc54-589d-4f19-96cf-10f6a0ac85ee None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:33.041122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc71615-3bf5-4d32-a279-9ebb7d291038 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:33.075539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eedf26-e46d-401d-8135-f8f9394cce1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:33.076189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eedf26-e46d-401d-8135-f8f9394cce1a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.076470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eedf26-e46d-401d-8135-f8f9394cce1a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.076740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eedf26-e46d-401d-8135-f8f9394cce1a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.082296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eedf26-e46d-401d-8135-f8f9394cce1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:33.083753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eedf26-e46d-401d-8135-f8f9394cce1a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:33.085361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 53/102] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:33.094012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eedf26-e46d-401d-8135-f8f9394cce1a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:33.162346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:33.235576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc71615-3bf5-4d32-a279-9ebb7d291038 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:33.257955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:33.286080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e51a324-d066-4f2c-93ca-be93fbd234ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:33.287124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51a324-d066-4f2c-93ca-be93fbd234ed None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.287411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51a324-d066-4f2c-93ca-be93fbd234ed None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.287688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51a324-d066-4f2c-93ca-be93fbd234ed None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.295710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eedf26-e46d-401d-8135-f8f9394cce1a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:33.298849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e51a324-d066-4f2c-93ca-be93fbd234ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-9e51a324-d066-4f2c-93ca-be93fbd234ed None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:33.304660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:33.305507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:33.306343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 50/103] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:52:33.307971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07007cd5-04b1-4d18-8fc3-b7e58f143c34 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.307971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07007cd5-04b1-4d18-8fc3-b7e58f143c34 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.307971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07007cd5-04b1-4d18-8fc3-b7e58f143c34 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.309939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 51/104] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:33.315200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07007cd5-04b1-4d18-8fc3-b7e58f143c34 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:33.400949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4dd0a4-44e9-47be-b0ed-2a79b04d2d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:33.401568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4dd0a4-44e9-47be-b0ed-2a79b04d2d21 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.401879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4dd0a4-44e9-47be-b0ed-2a79b04d2d21 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.402157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4dd0a4-44e9-47be-b0ed-2a79b04d2d21 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.403345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4dd0a4-44e9-47be-b0ed-2a79b04d2d21 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:33.404788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4dd0a4-44e9-47be-b0ed-2a79b04d2d21 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:33.449280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07007cd5-04b1-4d18-8fc3-b7e58f143c34 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:33.486776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242e8066-de8c-4209-9123-1ebed3c63516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:33.487394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e8066-de8c-4209-9123-1ebed3c63516 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.487679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e8066-de8c-4209-9123-1ebed3c63516 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.487954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e8066-de8c-4209-9123-1ebed3c63516 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.488615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e8066-de8c-4209-9123-1ebed3c63516 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:33.488921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e8066-de8c-4209-9123-1ebed3c63516 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:33.490468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e8066-de8c-4209-9123-1ebed3c63516 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:33.501571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 52/105] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:33.503335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d8b6c4-60ad-4cca-b554-b1ed5b79bb91 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.503828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d8b6c4-60ad-4cca-b554-b1ed5b79bb91 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.504305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d8b6c4-60ad-4cca-b554-b1ed5b79bb91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.504838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 54/106] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:52:33 2022] POST /identity/v3/projects => generated 302 bytes in 411 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:52:33.514682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191764c7-50c0-4536-a61c-c4a7dc19074a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.515335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191764c7-50c0-4536-a61c-c4a7dc19074a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.516549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191764c7-50c0-4536-a61c-c4a7dc19074a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.520239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 55/107] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:33 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:33.525727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191764c7-50c0-4536-a61c-c4a7dc19074a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:33.545356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-d9d8b6c4-60ad-4cca-b554-b1ed5b79bb91 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:33.546928 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d9d8b6c4-60ad-4cca-b554-b1ed5b79bb91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:33.583064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:33.677905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 53/108] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:33.679551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e4fcd-8d3c-4ba1-a582-6669e4cd59b7 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.681278 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e4fcd-8d3c-4ba1-a582-6669e4cd59b7 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.681645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e4fcd-8d3c-4ba1-a582-6669e4cd59b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.710439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-da4e4fcd-8d3c-4ba1-a582-6669e4cd59b7 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:33.712418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-da4e4fcd-8d3c-4ba1-a582-6669e4cd59b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:33.724856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:33.740091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191764c7-50c0-4536-a61c-c4a7dc19074a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:33.785347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae4e6a5-4325-412e-9337-d240e697bfe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:33.786324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae4e6a5-4325-412e-9337-d240e697bfe7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:33.787583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae4e6a5-4325-412e-9337-d240e697bfe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:33.788727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae4e6a5-4325-412e-9337-d240e697bfe7 None admin] PATH_INFO: `/v3/users/7fe723ad02f64965987bfb9686f25da9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:33.817133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae4e6a5-4325-412e-9337-d240e697bfe7 None admin] RBAC: Authorizing `identity:get_user(user_id=7fe723ad02f64965987bfb9686f25da9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:33.819661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae4e6a5-4325-412e-9337-d240e697bfe7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:33.821757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 56/109] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:33 2022] GET /identity/v3/users/7fe723ad02f64965987bfb9686f25da9 => generated 289 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:33.828758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae4e6a5-4325-412e-9337-d240e697bfe7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:33.844260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 54/110] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:33.942605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae4e6a5-4325-412e-9337-d240e697bfe7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:33.999291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc6bd08-af2d-4d13-ab4a-1d5c7193a1d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:33.999839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc6bd08-af2d-4d13-ab4a-1d5c7193a1d1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.000123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc6bd08-af2d-4d13-ab4a-1d5c7193a1d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.000396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc6bd08-af2d-4d13-ab4a-1d5c7193a1d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.024972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:34.031060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4e4fcd-8d3c-4ba1-a582-6669e4cd59b7 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:34.146473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc6bd08-af2d-4d13-ab4a-1d5c7193a1d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:34.147957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc6bd08-af2d-4d13-ab4a-1d5c7193a1d1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:34.224618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:34.247525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4e4fcd-8d3c-4ba1-a582-6669e4cd59b7 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:34.296574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42f377-5214-45a3-98a4-922d320890bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:34.297169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42f377-5214-45a3-98a4-922d320890bb None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.297413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42f377-5214-45a3-98a4-922d320890bb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.297723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42f377-5214-45a3-98a4-922d320890bb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.308467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae42f377-5214-45a3-98a4-922d320890bb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ae42f377-5214-45a3-98a4-922d320890bb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:34.313315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:34.314184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:34.314956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 55/111] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:52:34.315580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42f377-5214-45a3-98a4-922d320890bb None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:34.335439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 57/112] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:34.336628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48e7ca6-d2b6-4b61-bcf9-93aaac1989f8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.336919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48e7ca6-d2b6-4b61-bcf9-93aaac1989f8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.337449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48e7ca6-d2b6-4b61-bcf9-93aaac1989f8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.338916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 58/113] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:34.342824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982dbab-b4fc-4e60-a12a-ad401c274c58 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.343337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982dbab-b4fc-4e60-a12a-ad401c274c58 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.343845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5982dbab-b4fc-4e60-a12a-ad401c274c58 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.345634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 59/114] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:34.350362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5982dbab-b4fc-4e60-a12a-ad401c274c58 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:34.427592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42f377-5214-45a3-98a4-922d320890bb None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:34.428569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:34.463258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1385fc1-6103-4d35-94e3-2e4fc9a39e66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:34.463894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1385fc1-6103-4d35-94e3-2e4fc9a39e66 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.464186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1385fc1-6103-4d35-94e3-2e4fc9a39e66 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.464457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1385fc1-6103-4d35-94e3-2e4fc9a39e66 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.475224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1385fc1-6103-4d35-94e3-2e4fc9a39e66 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-b1385fc1-6103-4d35-94e3-2e4fc9a39e66 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:34.479997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:34.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:34.481924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 56/115] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:52:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:34.482573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000f7337-00a0-4972-9abb-f6331d7cc4b4 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.482637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000f7337-00a0-4972-9abb-f6331d7cc4b4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.482907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000f7337-00a0-4972-9abb-f6331d7cc4b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.500181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-000f7337-00a0-4972-9abb-f6331d7cc4b4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:34.500697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5982dbab-b4fc-4e60-a12a-ad401c274c58 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:34.501360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-000f7337-00a0-4972-9abb-f6331d7cc4b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:34.512671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:34.537631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:34.538264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.538533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.538815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.539556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:34.539867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:34.541234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:34.550161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 60/116] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:34.555278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:34.578683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 57/117] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:34.578683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5ee728-2f99-43ff-bc1d-8595d2261765 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.578683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5ee728-2f99-43ff-bc1d-8595d2261765 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.578683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5ee728-2f99-43ff-bc1d-8595d2261765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.592652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2f5ee728-2f99-43ff-bc1d-8595d2261765 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:34.593930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2f5ee728-2f99-43ff-bc1d-8595d2261765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:34.671030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac5e5c6-5722-4b3a-a11a-9c13502debb5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:34.671622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 58/118] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:34.672384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5ee728-2f99-43ff-bc1d-8595d2261765 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:34.695815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:34.750925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5abb15-fa40-4b55-91f2-e21dd6bbf0a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:34.751618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5abb15-fa40-4b55-91f2-e21dd6bbf0a9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.751903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5abb15-fa40-4b55-91f2-e21dd6bbf0a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.752175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5abb15-fa40-4b55-91f2-e21dd6bbf0a9 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.756696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5abb15-fa40-4b55-91f2-e21dd6bbf0a9 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:34.758338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5abb15-fa40-4b55-91f2-e21dd6bbf0a9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:34.758461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:34.760273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 61/119] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:34 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:34.761060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5abb15-fa40-4b55-91f2-e21dd6bbf0a9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:34.796468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5ee728-2f99-43ff-bc1d-8595d2261765 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:34.899230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:34.900202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.900708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.901179 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.902097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:34.903138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:34.903917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5abb15-fa40-4b55-91f2-e21dd6bbf0a9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:34.904966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:34.913748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 59/120] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:52:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:34.914785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:34.947768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc940c4-fc29-42c8-ac19-843fd29f552c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:34.948681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc940c4-fc29-42c8-ac19-843fd29f552c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:34.948976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc940c4-fc29-42c8-ac19-843fd29f552c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:34.949244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc940c4-fc29-42c8-ac19-843fd29f552c None admin] PATH_INFO: `/v3/users/c2010f49b97347fb8aff29794f27d93f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:34.950462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc940c4-fc29-42c8-ac19-843fd29f552c None admin] RBAC: Authorizing `identity:get_user(user_id=c2010f49b97347fb8aff29794f27d93f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:34.952931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc940c4-fc29-42c8-ac19-843fd29f552c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:34.954712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 62/121] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:34 2022] GET /identity/v3/users/c2010f49b97347fb8aff29794f27d93f => generated 247 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:34.958841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc940c4-fc29-42c8-ac19-843fd29f552c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.016708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:35.024954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b0150c-875e-4b79-bae0-3eec8c65beb3 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.067142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.067981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.068267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.068637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.069268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] RBAC: Adding query filter params (role.id=4ae9fa85613b4396ba3b5e4c3eff1cff, scope.project.id=8419cf9f31bb4753946ad654d60e9e26, user.id=7fe723ad02f64965987bfb9686f25da9) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:35.069622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:35.070680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc940c4-fc29-42c8-ac19-843fd29f552c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.071075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:35.075681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:35.078906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 60/122] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:34 2022] GET /identity/v3/role_assignments?user.id=7fe723ad02f64965987bfb9686f25da9&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=4ae9fa85613b4396ba3b5e4c3eff1cff => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:35.079554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.134991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542cb2da-7fb9-4adc-bf28-2a7b1641d0f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.136299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542cb2da-7fb9-4adc-bf28-2a7b1641d0f6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.136812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542cb2da-7fb9-4adc-bf28-2a7b1641d0f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.137802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542cb2da-7fb9-4adc-bf28-2a7b1641d0f6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.155802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542cb2da-7fb9-4adc-bf28-2a7b1641d0f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-542cb2da-7fb9-4adc-bf28-2a7b1641d0f6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:35.163482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:35.164419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:35.165184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:35.167414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 63/123] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:35.167414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542cb2da-7fb9-4adc-bf28-2a7b1641d0f6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.222236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9282ce-5cd3-4488-abc4-dd28adb5d49d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.266327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d0f8a5-61f5-4012-8c3e-d4cd01eb7fe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.267013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0f8a5-61f5-4012-8c3e-d4cd01eb7fe4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.267297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0f8a5-61f5-4012-8c3e-d4cd01eb7fe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.267573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0f8a5-61f5-4012-8c3e-d4cd01eb7fe4 None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.269097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d0f8a5-61f5-4012-8c3e-d4cd01eb7fe4 None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:35.271202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d0f8a5-61f5-4012-8c3e-d4cd01eb7fe4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:35.273156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 61/124] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:35 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:35.274343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d0f8a5-61f5-4012-8c3e-d4cd01eb7fe4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.296157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542cb2da-7fb9-4adc-bf28-2a7b1641d0f6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.305256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:35.342690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4ba536-e8cf-4c18-a052-51120ca94736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.343341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4ba536-e8cf-4c18-a052-51120ca94736 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.343631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4ba536-e8cf-4c18-a052-51120ca94736 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.343927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4ba536-e8cf-4c18-a052-51120ca94736 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.345230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4ba536-e8cf-4c18-a052-51120ca94736 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:35.346841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4ba536-e8cf-4c18-a052-51120ca94736 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:35.348585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 64/125] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:35 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:35.354316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4ba536-e8cf-4c18-a052-51120ca94736 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.357632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:35.462025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d0f8a5-61f5-4012-8c3e-d4cd01eb7fe4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.498954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3a9066-41f7-458d-979e-718046421399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.499753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3a9066-41f7-458d-979e-718046421399 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.500134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3a9066-41f7-458d-979e-718046421399 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.500562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3a9066-41f7-458d-979e-718046421399 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.511940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3a9066-41f7-458d-979e-718046421399 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-0f3a9066-41f7-458d-979e-718046421399 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:35.517426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:35.518497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:35.519459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:35.521073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 62/126] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:35.522021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3a9066-41f7-458d-979e-718046421399 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.555303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4ba536-e8cf-4c18-a052-51120ca94736 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.594665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.595364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.595752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.596134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.596965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:35.597422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:35.599098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:35.607945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 65/127] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:35.613345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.643007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3a9066-41f7-458d-979e-718046421399 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.670052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:35.676413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:35.687268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2cdbc8-bbfb-4292-82ff-9bfcf46ad280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.688152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2cdbc8-bbfb-4292-82ff-9bfcf46ad280 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.688447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2cdbc8-bbfb-4292-82ff-9bfcf46ad280 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.688728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2cdbc8-bbfb-4292-82ff-9bfcf46ad280 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.690157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2cdbc8-bbfb-4292-82ff-9bfcf46ad280 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:35.692447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2cdbc8-bbfb-4292-82ff-9bfcf46ad280 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:35.694133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 63/128] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:35 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:35.694908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2cdbc8-bbfb-4292-82ff-9bfcf46ad280 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.732921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aa3772-7c3f-43b7-94cc-da9f3cd8549a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.767930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a2cebe-bf25-49ff-a783-46eb46f12dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.768721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a2cebe-bf25-49ff-a783-46eb46f12dc2 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.769086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a2cebe-bf25-49ff-a783-46eb46f12dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.769366 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a2cebe-bf25-49ff-a783-46eb46f12dc2 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26/users/c2010f49b97347fb8aff29794f27d93f/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.771321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a2cebe-bf25-49ff-a783-46eb46f12dc2 None admin] RBAC: Authorizing `identity:create_grant(project_id=8419cf9f31bb4753946ad654d60e9e26, user_id=c2010f49b97347fb8aff29794f27d93f, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:35.772770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a2cebe-bf25-49ff-a783-46eb46f12dc2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:35.801297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2cdbc8-bbfb-4292-82ff-9bfcf46ad280 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:35.835039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:35.835648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:35.835930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:35.836204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:35.836857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:35.837426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:35.838848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:35.846708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 64/129] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:35.847482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.873857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 66/130] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:35 2022] PUT /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26/users/c2010f49b97347fb8aff29794f27d93f/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:35.874786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a2cebe-bf25-49ff-a783-46eb46f12dc2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:35.917028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:35.979263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355f73f4-5bad-4985-ab71-167d3f541600 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:36.026495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e99b681-7ce7-4191-9aa6-140090239c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:36.027360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e99b681-7ce7-4191-9aa6-140090239c83 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.027890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e99b681-7ce7-4191-9aa6-140090239c83 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.028316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e99b681-7ce7-4191-9aa6-140090239c83 None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.029620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e99b681-7ce7-4191-9aa6-140090239c83 None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:36.031271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e99b681-7ce7-4191-9aa6-140090239c83 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:36.032872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 65/131] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:35 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:36.032996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:36.036099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e77648d-6806-4c9b-be82-3d0aeb5eff9b None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.036099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e77648d-6806-4c9b-be82-3d0aeb5eff9b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.036099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e77648d-6806-4c9b-be82-3d0aeb5eff9b None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.036099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 66/132] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:36 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:36.036581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e77648d-6806-4c9b-be82-3d0aeb5eff9b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:36.045797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a2cebe-bf25-49ff-a783-46eb46f12dc2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:36.089290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff68eb4-564e-408e-8468-46c714ad6b28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:36.090208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff68eb4-564e-408e-8468-46c714ad6b28 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.090501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff68eb4-564e-408e-8468-46c714ad6b28 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.091210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff68eb4-564e-408e-8468-46c714ad6b28 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26/users/055c8932e5b14530ad56960260a63d7e/roles/609ec30d147941b4a2b5508980aea7d0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.093772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff68eb4-564e-408e-8468-46c714ad6b28 None admin] RBAC: Authorizing `identity:create_grant(project_id=8419cf9f31bb4753946ad654d60e9e26, user_id=055c8932e5b14530ad56960260a63d7e, role_id=609ec30d147941b4a2b5508980aea7d0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:36.095850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff68eb4-564e-408e-8468-46c714ad6b28 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:36.150274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e77648d-6806-4c9b-be82-3d0aeb5eff9b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:36.177469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:36.187237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17babe19-84c0-4b05-bc60-193f8e7b748e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:36.187892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17babe19-84c0-4b05-bc60-193f8e7b748e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.188174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17babe19-84c0-4b05-bc60-193f8e7b748e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.188445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17babe19-84c0-4b05-bc60-193f8e7b748e None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.189881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17babe19-84c0-4b05-bc60-193f8e7b748e None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:36.191330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17babe19-84c0-4b05-bc60-193f8e7b748e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:36.193013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 67/133] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:36 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:36.198431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec3362-30be-4a2c-9456-bbd4faa9e310 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.198648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 67/134] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:35 2022] PUT /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26/users/055c8932e5b14530ad56960260a63d7e/roles/609ec30d147941b4a2b5508980aea7d0 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:36.198748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec3362-30be-4a2c-9456-bbd4faa9e310 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.199037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec3362-30be-4a2c-9456-bbd4faa9e310 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.200078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff68eb4-564e-408e-8468-46c714ad6b28 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:36.223569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-06ec3362-30be-4a2c-9456-bbd4faa9e310 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:36.227007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-06ec3362-30be-4a2c-9456-bbd4faa9e310 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:36.304525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 68/135] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:36.305613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ec3362-30be-4a2c-9456-bbd4faa9e310 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:36.354915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff68eb4-564e-408e-8468-46c714ad6b28 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:36.397611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c97cee-8ec3-4e9a-a211-de1cbdb10046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:36.398686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c97cee-8ec3-4e9a-a211-de1cbdb10046 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.399217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c97cee-8ec3-4e9a-a211-de1cbdb10046 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.399637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c97cee-8ec3-4e9a-a211-de1cbdb10046 None admin] PATH_INFO: `/v3/projects/5492b71bdd6c44b6b273375d9450b759` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.401298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c97cee-8ec3-4e9a-a211-de1cbdb10046 None admin] RBAC: Authorizing `identity:get_project(project_id=5492b71bdd6c44b6b273375d9450b759)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:36.403483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c97cee-8ec3-4e9a-a211-de1cbdb10046 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:36.405373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 68/136] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:36 2022] GET /identity/v3/projects/5492b71bdd6c44b6b273375d9450b759 => generated 312 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:36.414685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c97cee-8ec3-4e9a-a211-de1cbdb10046 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:36.477097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:36.507523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ec3362-30be-4a2c-9456-bbd4faa9e310 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:36.557436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:36.579794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf63d063-763e-4c38-b9d4-0f8eeed39485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:36.580448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63d063-763e-4c38-b9d4-0f8eeed39485 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.580733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63d063-763e-4c38-b9d4-0f8eeed39485 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.581012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf63d063-763e-4c38-b9d4-0f8eeed39485 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.583274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf63d063-763e-4c38-b9d4-0f8eeed39485 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:36.584817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf63d063-763e-4c38-b9d4-0f8eeed39485 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:36.587716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 69/137] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:36 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:36.593783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf63d063-763e-4c38-b9d4-0f8eeed39485 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:36.637993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c97cee-8ec3-4e9a-a211-de1cbdb10046 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:36.704857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ac01d2-7b1e-45ae-8e84-0d9d9113ffa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:36.705580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ac01d2-7b1e-45ae-8e84-0d9d9113ffa4 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.706099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ac01d2-7b1e-45ae-8e84-0d9d9113ffa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.706389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ac01d2-7b1e-45ae-8e84-0d9d9113ffa4 None admin] PATH_INFO: `/v3/projects/5492b71bdd6c44b6b273375d9450b759/users/055c8932e5b14530ad56960260a63d7e/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.708847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ac01d2-7b1e-45ae-8e84-0d9d9113ffa4 None admin] RBAC: Authorizing `identity:create_grant(project_id=5492b71bdd6c44b6b273375d9450b759, user_id=055c8932e5b14530ad56960260a63d7e, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:36.710324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ac01d2-7b1e-45ae-8e84-0d9d9113ffa4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:36.761424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf63d063-763e-4c38-b9d4-0f8eeed39485 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:36.790240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 69/138] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:36 2022] PUT /identity/v3/projects/5492b71bdd6c44b6b273375d9450b759/users/055c8932e5b14530ad56960260a63d7e/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:36.791521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ac01d2-7b1e-45ae-8e84-0d9d9113ffa4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:36.810981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009660e0-cb00-43b3-aa56-c69a9e9e79ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:36.811605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009660e0-cb00-43b3-aa56-c69a9e9e79ec None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:36.811886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009660e0-cb00-43b3-aa56-c69a9e9e79ec None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:36.812158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009660e0-cb00-43b3-aa56-c69a9e9e79ec None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26/users/055c8932e5b14530ad56960260a63d7e/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:36.819166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009660e0-cb00-43b3-aa56-c69a9e9e79ec None admin] RBAC: Authorizing `identity:create_grant(project_id=8419cf9f31bb4753946ad654d60e9e26, user_id=055c8932e5b14530ad56960260a63d7e, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:36.820661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009660e0-cb00-43b3-aa56-c69a9e9e79ec None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:36.903622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 70/139] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:36 2022] PUT /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26/users/055c8932e5b14530ad56960260a63d7e/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:36.925112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:36.949830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ac01d2-7b1e-45ae-8e84-0d9d9113ffa4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:37.001475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b9e3bc-815c-49d5-a378-e0ff8ccefdec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:37.002213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9e3bc-815c-49d5-a378-e0ff8ccefdec None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:37.002557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9e3bc-815c-49d5-a378-e0ff8ccefdec None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:37.002840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b9e3bc-815c-49d5-a378-e0ff8ccefdec None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:37.004048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b9e3bc-815c-49d5-a378-e0ff8ccefdec None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:37.005544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b9e3bc-815c-49d5-a378-e0ff8ccefdec None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:37.007428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 70/140] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:37.012895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009660e0-cb00-43b3-aa56-c69a9e9e79ec None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:37.108196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:37.174261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009660e0-cb00-43b3-aa56-c69a9e9e79ec None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:37.217296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d58fc-3394-4d07-bbba-75a0b0326574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:37.220337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d58fc-3394-4d07-bbba-75a0b0326574 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:37.220337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d58fc-3394-4d07-bbba-75a0b0326574 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:37.220337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d58fc-3394-4d07-bbba-75a0b0326574 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:37.220337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9d58fc-3394-4d07-bbba-75a0b0326574 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:37.220728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9d58fc-3394-4d07-bbba-75a0b0326574 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:37.352862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 71/141] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:52:37 2022] POST /identity/v3/users => generated 283 bytes in 340 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:37.372188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b64e4-1f88-438f-bd0a-ecd7d4ff920f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:37.372188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b64e4-1f88-438f-bd0a-ecd7d4ff920f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:37.372188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b64e4-1f88-438f-bd0a-ecd7d4ff920f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:37.373725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 71/142] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:37.384295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0a279-ae2f-4e50-993b-8688cf1a4c7e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:37.384648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0a279-ae2f-4e50-993b-8688cf1a4c7e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:37.384930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd0a279-ae2f-4e50-993b-8688cf1a4c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:37.411152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3fd0a279-ae2f-4e50-993b-8688cf1a4c7e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:37.412444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3fd0a279-ae2f-4e50-993b-8688cf1a4c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:37.498810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 72/143] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:37.950934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67b64e4-1f88-438f-bd0a-ecd7d4ff920f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:38.057511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67b64e4-1f88-438f-bd0a-ecd7d4ff920f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:38.095631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db13bafe-14eb-434b-8f55-783e9355f4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:38.096276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db13bafe-14eb-434b-8f55-783e9355f4e8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:38.096618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db13bafe-14eb-434b-8f55-783e9355f4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:38.096902 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db13bafe-14eb-434b-8f55-783e9355f4e8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:38.108123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db13bafe-14eb-434b-8f55-783e9355f4e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-db13bafe-14eb-434b-8f55-783e9355f4e8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:38.113019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:38.113919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:38.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 72/144] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:38.120028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd0a279-ae2f-4e50-993b-8688cf1a4c7e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:38.130071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558150d-2ece-4189-8ffe-37f3407fb80c None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:38.130358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558150d-2ece-4189-8ffe-37f3407fb80c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:38.130643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558150d-2ece-4189-8ffe-37f3407fb80c None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:38.131982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 73/145] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:38.138097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7b138-033f-4eb9-bcdc-65b1c62ce4f9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:38.138381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7b138-033f-4eb9-bcdc-65b1c62ce4f9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:38.138667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb7b138-033f-4eb9-bcdc-65b1c62ce4f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:38.174308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8eb7b138-033f-4eb9-bcdc-65b1c62ce4f9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:38.175530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8eb7b138-033f-4eb9-bcdc-65b1c62ce4f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:38.181436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:38.190155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:38.287953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 74/146] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:38.319808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd0a279-ae2f-4e50-993b-8688cf1a4c7e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:38.371333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:38.372065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:38.372359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:38.372636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:38.373319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:38.373652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:38.375056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:38.389819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 73/147] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:38.396577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb7b138-033f-4eb9-bcdc-65b1c62ce4f9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:38.538612 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb7b138-033f-4eb9-bcdc-65b1c62ce4f9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:38.580657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a97c11-5e9e-474e-9539-60b8db83c0ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:38.581367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a97c11-5e9e-474e-9539-60b8db83c0ad None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:38.581676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a97c11-5e9e-474e-9539-60b8db83c0ad None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:38.581953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a97c11-5e9e-474e-9539-60b8db83c0ad None admin] PATH_INFO: `/v3/users/7fe723ad02f64965987bfb9686f25da9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:38.583099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a97c11-5e9e-474e-9539-60b8db83c0ad None admin] RBAC: Authorizing `identity:get_user(user_id=7fe723ad02f64965987bfb9686f25da9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:38.584538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a97c11-5e9e-474e-9539-60b8db83c0ad None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:38.590759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 75/148] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:38 2022] GET /identity/v3/users/7fe723ad02f64965987bfb9686f25da9 => generated 289 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:38.596564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:38.745411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a97c11-5e9e-474e-9539-60b8db83c0ad None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:38.753536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:38.757113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b8e405-b7c9-41a2-b88f-bc76d410699a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:38.758733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:38.811931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25297d88-5627-499f-8e8e-cd7bd1c315d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:38.812535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25297d88-5627-499f-8e8e-cd7bd1c315d5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:38.812821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25297d88-5627-499f-8e8e-cd7bd1c315d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:38.813096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25297d88-5627-499f-8e8e-cd7bd1c315d5 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:38.818377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25297d88-5627-499f-8e8e-cd7bd1c315d5 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:38.820238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25297d88-5627-499f-8e8e-cd7bd1c315d5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:38.824756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 74/149] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:38 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:38.841203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25297d88-5627-499f-8e8e-cd7bd1c315d5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:38.862663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a97c11-5e9e-474e-9539-60b8db83c0ad None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:38.930323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ad6f58-0ba5-4cbc-bf60-d087748f440b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:38.931073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad6f58-0ba5-4cbc-bf60-d087748f440b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:38.931357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad6f58-0ba5-4cbc-bf60-d087748f440b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:38.931638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ad6f58-0ba5-4cbc-bf60-d087748f440b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:38.958499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ad6f58-0ba5-4cbc-bf60-d087748f440b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-62ad6f58-0ba5-4cbc-bf60-d087748f440b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:38.964292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:38.965196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:38.966015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 76/150] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:52:38.974037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ad6f58-0ba5-4cbc-bf60-d087748f440b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:39.039995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25297d88-5627-499f-8e8e-cd7bd1c315d5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:39.104055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fff2bda-d323-4fdf-aedf-c295f3fca5a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:39.104661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff2bda-d323-4fdf-aedf-c295f3fca5a0 None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:39.105073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff2bda-d323-4fdf-aedf-c295f3fca5a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:39.105352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fff2bda-d323-4fdf-aedf-c295f3fca5a0 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26/users/7fe723ad02f64965987bfb9686f25da9/roles/4ae9fa85613b4396ba3b5e4c3eff1cff` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:39.107958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fff2bda-d323-4fdf-aedf-c295f3fca5a0 None admin] RBAC: Authorizing `identity:create_grant(project_id=8419cf9f31bb4753946ad654d60e9e26, user_id=7fe723ad02f64965987bfb9686f25da9, role_id=4ae9fa85613b4396ba3b5e4c3eff1cff)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:39.109408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fff2bda-d323-4fdf-aedf-c295f3fca5a0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:39.128765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:39.177926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ad6f58-0ba5-4cbc-bf60-d087748f440b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:39.201978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 75/151] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:38 2022] PUT /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26/users/7fe723ad02f64965987bfb9686f25da9/roles/4ae9fa85613b4396ba3b5e4c3eff1cff => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 04:52:39.206783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319e1692-c85a-41ff-bb52-b3ff549cb588 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:39.206783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319e1692-c85a-41ff-bb52-b3ff549cb588 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:39.206783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319e1692-c85a-41ff-bb52-b3ff549cb588 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:39.208074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 76/152] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:39.213819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067127b3-ad0a-4606-b747-871b32d565dd None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:39.216175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067127b3-ad0a-4606-b747-871b32d565dd None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:39.216569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067127b3-ad0a-4606-b747-871b32d565dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:39.225099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:39.225862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:39.226099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:39.226332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:39.227096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:39.227403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:39.229009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:39.245264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 77/153] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:39.253049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:39.265462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-067127b3-ad0a-4606-b747-871b32d565dd None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:39.266725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-067127b3-ad0a-4606-b747-871b32d565dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:39.306355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:39.387283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 77/154] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:39.467389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d62eae9-7e4c-44e0-998b-38547583b54a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:39.519208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:39.525120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf23a203-8053-494f-8c15-f8f8aeeb636f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:39.525813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf23a203-8053-494f-8c15-f8f8aeeb636f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:39.526097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf23a203-8053-494f-8c15-f8f8aeeb636f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:39.526374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf23a203-8053-494f-8c15-f8f8aeeb636f None admin] PATH_INFO: `/v3/users/c2010f49b97347fb8aff29794f27d93f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:39.527908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf23a203-8053-494f-8c15-f8f8aeeb636f None admin] RBAC: Authorizing `identity:get_user(user_id=c2010f49b97347fb8aff29794f27d93f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:39.529407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf23a203-8053-494f-8c15-f8f8aeeb636f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:39.531400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 78/155] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:39 2022] GET /identity/v3/users/c2010f49b97347fb8aff29794f27d93f => generated 247 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:39.537187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067127b3-ad0a-4606-b747-871b32d565dd None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:39.702759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067127b3-ad0a-4606-b747-871b32d565dd None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:39.744528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3fc4df-8944-45e9-9f35-551a4676268a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:39.745147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3fc4df-8944-45e9-9f35-551a4676268a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:39.745423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3fc4df-8944-45e9-9f35-551a4676268a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:39.745719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3fc4df-8944-45e9-9f35-551a4676268a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:39.777529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:39.826836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3fc4df-8944-45e9-9f35-551a4676268a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:39.828317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3fc4df-8944-45e9-9f35-551a4676268a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:39.934221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 78/156] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:39.940215 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf23a203-8053-494f-8c15-f8f8aeeb636f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:40.034034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3fc4df-8944-45e9-9f35-551a4676268a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:40.072044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf23a203-8053-494f-8c15-f8f8aeeb636f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:40.117319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4d4790-d83f-4303-b72f-75c94fabd8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:40.118338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d4790-d83f-4303-b72f-75c94fabd8fe None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.120010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d4790-d83f-4303-b72f-75c94fabd8fe None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.120565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d4790-d83f-4303-b72f-75c94fabd8fe None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.123319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4d4790-d83f-4303-b72f-75c94fabd8fe None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:40.126530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4d4790-d83f-4303-b72f-75c94fabd8fe None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:40.129419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 79/157] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:39 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:40.130841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1193bf3-3831-46ce-8957-8df6b8eb5893 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.131382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1193bf3-3831-46ce-8957-8df6b8eb5893 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.131850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1193bf3-3831-46ce-8957-8df6b8eb5893 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.137760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 80/158] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:40 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:40.139986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1193bf3-3831-46ce-8957-8df6b8eb5893 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:40.156165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:40.256397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:40.278385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3fc4df-8944-45e9-9f35-551a4676268a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:40.353896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1193bf3-3831-46ce-8957-8df6b8eb5893 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:40.363934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae97640e-b301-475f-a567-564ec7175b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:40.369039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae97640e-b301-475f-a567-564ec7175b29 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.369577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae97640e-b301-475f-a567-564ec7175b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.370084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae97640e-b301-475f-a567-564ec7175b29 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.399315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae97640e-b301-475f-a567-564ec7175b29 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ae97640e-b301-475f-a567-564ec7175b29 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:40.414734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:40.415599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:40.416436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:40.421771 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 79/159] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:52:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 388 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:40.423812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c016837-b9eb-4fc5-bd2b-eec92cb64bc2 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.424459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c016837-b9eb-4fc5-bd2b-eec92cb64bc2 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.426119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c016837-b9eb-4fc5-bd2b-eec92cb64bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.441105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810ee182-5334-43ba-af05-320c6df07dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:40.445124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810ee182-5334-43ba-af05-320c6df07dff None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.445641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810ee182-5334-43ba-af05-320c6df07dff None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.446223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810ee182-5334-43ba-af05-320c6df07dff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.447207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810ee182-5334-43ba-af05-320c6df07dff None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=8419cf9f31bb4753946ad654d60e9e26, user.id=c2010f49b97347fb8aff29794f27d93f) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:40.447842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810ee182-5334-43ba-af05-320c6df07dff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:40.450015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810ee182-5334-43ba-af05-320c6df07dff None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:40.460436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1c016837-b9eb-4fc5-bd2b-eec92cb64bc2 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:40.462333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1c016837-b9eb-4fc5-bd2b-eec92cb64bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:40.464656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 81/160] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:40 2022] GET /identity/v3/role_assignments?user.id=c2010f49b97347fb8aff29794f27d93f&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 608 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:40.466190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9071c97-d34e-4269-8562-ea24c97bd75f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.468390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9071c97-d34e-4269-8562-ea24c97bd75f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.469305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9071c97-d34e-4269-8562-ea24c97bd75f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.470940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 82/161] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:40.472560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9071c97-d34e-4269-8562-ea24c97bd75f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:40.623689 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 80/162] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:40.630338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdfea04-66fd-4b9b-8ef4-0a8c1c93c429 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.630622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdfea04-66fd-4b9b-8ef4-0a8c1c93c429 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.630892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdfea04-66fd-4b9b-8ef4-0a8c1c93c429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.657567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-bcdfea04-66fd-4b9b-8ef4-0a8c1c93c429 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:40.658769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-bcdfea04-66fd-4b9b-8ef4-0a8c1c93c429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:40.722787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9071c97-d34e-4269-8562-ea24c97bd75f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:40.723400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:40.764497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 81/163] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:40.776437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:40.788559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:40.789556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.790074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.790532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.791460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:40.791949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:40.793766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:40.810956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 83/164] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:52:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:40.816472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdfea04-66fd-4b9b-8ef4-0a8c1c93c429 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:40.935830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdfea04-66fd-4b9b-8ef4-0a8c1c93c429 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:40.975247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3822cbcb-faa1-4092-96fe-a569982ca27f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:40.975886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822cbcb-faa1-4092-96fe-a569982ca27f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:40.976159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822cbcb-faa1-4092-96fe-a569982ca27f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:40.976429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822cbcb-faa1-4092-96fe-a569982ca27f None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:40.977559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3822cbcb-faa1-4092-96fe-a569982ca27f None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:40.979022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3822cbcb-faa1-4092-96fe-a569982ca27f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:40.980660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 82/165] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:40 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:40.986292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:41.125129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7f8452-b4ef-4eec-b4c7-32376607f571 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:41.164631 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3822cbcb-faa1-4092-96fe-a569982ca27f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:41.170533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e857a843-9067-4555-8134-c51cbe38bd8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:41.171549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e857a843-9067-4555-8134-c51cbe38bd8c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:41.172078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e857a843-9067-4555-8134-c51cbe38bd8c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:41.172544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e857a843-9067-4555-8134-c51cbe38bd8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:41.257334 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:41.285570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:41.302570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e857a843-9067-4555-8134-c51cbe38bd8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:41.304083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e857a843-9067-4555-8134-c51cbe38bd8c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:41.356090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3822cbcb-faa1-4092-96fe-a569982ca27f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:41.422522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701187fe-47a9-4e0d-b951-654984d49267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:41.423168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701187fe-47a9-4e0d-b951-654984d49267 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:41.423447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701187fe-47a9-4e0d-b951-654984d49267 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:41.423717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701187fe-47a9-4e0d-b951-654984d49267 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:41.451219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701187fe-47a9-4e0d-b951-654984d49267 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-701187fe-47a9-4e0d-b951-654984d49267 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:41.456505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:41.457349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:41.458398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 83/166] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:52:41.463318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701187fe-47a9-4e0d-b951-654984d49267 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:41.502586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 84/167] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 517 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:41.506351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e857a843-9067-4555-8134-c51cbe38bd8c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:41.682713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701187fe-47a9-4e0d-b951-654984d49267 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:41.693923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:41.705958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:41.722207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e857a843-9067-4555-8134-c51cbe38bd8c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:41.738153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcccfa69-a080-420a-a480-489b62c78205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:41.740106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcccfa69-a080-420a-a480-489b62c78205 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:41.740552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcccfa69-a080-420a-a480-489b62c78205 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:41.740963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcccfa69-a080-420a-a480-489b62c78205 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:41.760668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcccfa69-a080-420a-a480-489b62c78205 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-fcccfa69-a080-420a-a480-489b62c78205 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:41.766765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:41.767845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:41.768913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:41.771002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 84/168] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:41.778620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcccfa69-a080-420a-a480-489b62c78205 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:41.799223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d906f1d5-eafd-419c-b6cb-56da24615b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:41.799836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d906f1d5-eafd-419c-b6cb-56da24615b80 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:41.800110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d906f1d5-eafd-419c-b6cb-56da24615b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:41.800376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d906f1d5-eafd-419c-b6cb-56da24615b80 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:41.801100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d906f1d5-eafd-419c-b6cb-56da24615b80 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:41.801505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d906f1d5-eafd-419c-b6cb-56da24615b80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:41.803017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d906f1d5-eafd-419c-b6cb-56da24615b80 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:41.820116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 85/169] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:41.821004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0c921e-a253-4cb9-8949-6ade7ebb490c None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:41.821289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0c921e-a253-4cb9-8949-6ade7ebb490c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:41.821560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0c921e-a253-4cb9-8949-6ade7ebb490c None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:41.823041 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 86/170] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:41.823576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0c921e-a253-4cb9-8949-6ade7ebb490c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:41.921729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcccfa69-a080-420a-a480-489b62c78205 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:41.959555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56871634-705b-414f-a66e-c0c201df4084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:41.960203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56871634-705b-414f-a66e-c0c201df4084 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:41.960482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56871634-705b-414f-a66e-c0c201df4084 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:41.960752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56871634-705b-414f-a66e-c0c201df4084 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:41.961932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56871634-705b-414f-a66e-c0c201df4084 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:41.963961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56871634-705b-414f-a66e-c0c201df4084 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:41.966543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 85/171] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:41 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:41.967766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46eb18-dba7-4ba6-aa6d-76eb9230cdd4 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:41.968054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46eb18-dba7-4ba6-aa6d-76eb9230cdd4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:41.968324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46eb18-dba7-4ba6-aa6d-76eb9230cdd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.001650 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3f46eb18-dba7-4ba6-aa6d-76eb9230cdd4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:42.002849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3f46eb18-dba7-4ba6-aa6d-76eb9230cdd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:42.016672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0c921e-a253-4cb9-8949-6ade7ebb490c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:42.055993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:42.056651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.057542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.057839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.058601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:42.058906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:42.060327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:42.069667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 87/172] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:42.070527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:42.083007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:42.097931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:42.178247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 86/173] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:42.179169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f46eb18-dba7-4ba6-aa6d-76eb9230cdd4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:42.192250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c40b648-d2ff-4e62-9ff4-b5dfc588c3bc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:42.239973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67f8290-a427-4de2-b86a-b0b3bb1df863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:42.240618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f8290-a427-4de2-b86a-b0b3bb1df863 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.240894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f8290-a427-4de2-b86a-b0b3bb1df863 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.241165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f8290-a427-4de2-b86a-b0b3bb1df863 None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.242323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67f8290-a427-4de2-b86a-b0b3bb1df863 None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:42.243897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67f8290-a427-4de2-b86a-b0b3bb1df863 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:42.246432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 88/174] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:42 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:42.249623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67f8290-a427-4de2-b86a-b0b3bb1df863 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:42.338925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:42.357112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67f8290-a427-4de2-b86a-b0b3bb1df863 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:42.395811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7808d2-5cd7-4f4e-8717-cf10ae31c7cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:42.396450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7808d2-5cd7-4f4e-8717-cf10ae31c7cf None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.396729 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7808d2-5cd7-4f4e-8717-cf10ae31c7cf None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.397043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7808d2-5cd7-4f4e-8717-cf10ae31c7cf None admin] PATH_INFO: `/v3/users/055c8932e5b14530ad56960260a63d7e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.398185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7808d2-5cd7-4f4e-8717-cf10ae31c7cf None admin] RBAC: Authorizing `identity:get_user(user_id=055c8932e5b14530ad56960260a63d7e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:42.399413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f46eb18-dba7-4ba6-aa6d-76eb9230cdd4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:42.399853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7808d2-5cd7-4f4e-8717-cf10ae31c7cf None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:42.401228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 89/175] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:42 2022] GET /identity/v3/users/055c8932e5b14530ad56960260a63d7e => generated 275 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:42.405621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfd060-d0d7-4ebb-984d-316d55065fa3 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.405621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfd060-d0d7-4ebb-984d-316d55065fa3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.405621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfd060-d0d7-4ebb-984d-316d55065fa3 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.405621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 90/176] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:42.405621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdfd060-d0d7-4ebb-984d-316d55065fa3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:42.442660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d8d712-00b7-427d-a565-779e7c51f2d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:42.443262 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d8d712-00b7-427d-a565-779e7c51f2d7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.443537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d8d712-00b7-427d-a565-779e7c51f2d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.443804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d8d712-00b7-427d-a565-779e7c51f2d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.444529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d8d712-00b7-427d-a565-779e7c51f2d7 None admin] RBAC: Adding query filter params (role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=8419cf9f31bb4753946ad654d60e9e26, user.id=055c8932e5b14530ad56960260a63d7e) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:42.444828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d8d712-00b7-427d-a565-779e7c51f2d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:42.446301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d8d712-00b7-427d-a565-779e7c51f2d7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:42.452564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 87/177] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:42 2022] GET /identity/v3/role_assignments?user.id=055c8932e5b14530ad56960260a63d7e&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 608 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:42.454030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d3ba7-aabe-4951-92c9-ddd8c79b0dd3 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.454308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d3ba7-aabe-4951-92c9-ddd8c79b0dd3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.454578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d3ba7-aabe-4951-92c9-ddd8c79b0dd3 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.455844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 88/178] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:42.457127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a8259-ba38-47d1-8f9d-9f32fa7eec00 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.457408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a8259-ba38-47d1-8f9d-9f32fa7eec00 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.457715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a8259-ba38-47d1-8f9d-9f32fa7eec00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.478737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-124a8259-ba38-47d1-8f9d-9f32fa7eec00 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:42.479875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-124a8259-ba38-47d1-8f9d-9f32fa7eec00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:42.522609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:42.526295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdfd060-d0d7-4ebb-984d-316d55065fa3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:42.600858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9150159-893a-4b35-a146-340677d194c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:42.600858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9150159-893a-4b35-a146-340677d194c6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.600858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9150159-893a-4b35-a146-340677d194c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.600858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9150159-893a-4b35-a146-340677d194c6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.631861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:42.658873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 89/179] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:42.659741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124a8259-ba38-47d1-8f9d-9f32fa7eec00 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:42.705847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9150159-893a-4b35-a146-340677d194c6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:42.707647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9150159-893a-4b35-a146-340677d194c6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:42.807970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124a8259-ba38-47d1-8f9d-9f32fa7eec00 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:42.832010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 91/180] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:42.833606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6ab7a5-c0fa-4f2c-a7e8-ec6dc63c24cd None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.834089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6ab7a5-c0fa-4f2c-a7e8-ec6dc63c24cd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.834540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6ab7a5-c0fa-4f2c-a7e8-ec6dc63c24cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.853688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1c6ab7a5-c0fa-4f2c-a7e8-ec6dc63c24cd None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:42.854760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1c6ab7a5-c0fa-4f2c-a7e8-ec6dc63c24cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:42.856077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23e4bdd-9d20-45b9-b248-4810509e9ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:42.857173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23e4bdd-9d20-45b9-b248-4810509e9ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:42.858533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23e4bdd-9d20-45b9-b248-4810509e9ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:42.858988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23e4bdd-9d20-45b9-b248-4810509e9ab5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:42.903434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:42.909357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:42.936149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 92/181] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:42.937131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6ab7a5-c0fa-4f2c-a7e8-ec6dc63c24cd None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:42.960896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23e4bdd-9d20-45b9-b248-4810509e9ab5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:42.964182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23e4bdd-9d20-45b9-b248-4810509e9ab5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:43.062222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6ab7a5-c0fa-4f2c-a7e8-ec6dc63c24cd None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.070194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 90/182] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:43.072102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc2c2f-79c4-4699-90b7-f46a68778603 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.072614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc2c2f-79c4-4699-90b7-f46a68778603 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.073358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc2c2f-79c4-4699-90b7-f46a68778603 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.075165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 91/183] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:43.076680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bc2c2f-79c4-4699-90b7-f46a68778603 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.098498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab1a29-d601-44f9-84fc-c11272c44299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.099140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab1a29-d601-44f9-84fc-c11272c44299 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.099418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab1a29-d601-44f9-84fc-c11272c44299 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.099688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab1a29-d601-44f9-84fc-c11272c44299 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.112240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ab1a29-d601-44f9-84fc-c11272c44299 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-32ab1a29-d601-44f9-84fc-c11272c44299 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:43.116949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:43.117854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:43.118886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 93/184] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:43.124092 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab1a29-d601-44f9-84fc-c11272c44299 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.141303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:43.171354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:43.184975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bc2c2f-79c4-4699-90b7-f46a68778603 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.221058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9921b583-7602-4b50-964e-35188ef6e2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.221738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9921b583-7602-4b50-964e-35188ef6e2f6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.222022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9921b583-7602-4b50-964e-35188ef6e2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.222297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9921b583-7602-4b50-964e-35188ef6e2f6 None admin] PATH_INFO: `/v3/projects/5492b71bdd6c44b6b273375d9450b759` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.223475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9921b583-7602-4b50-964e-35188ef6e2f6 None admin] RBAC: Authorizing `identity:get_project(project_id=5492b71bdd6c44b6b273375d9450b759)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:43.226057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9921b583-7602-4b50-964e-35188ef6e2f6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:43.226588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 92/185] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/projects/5492b71bdd6c44b6b273375d9450b759 => generated 312 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:43.227345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9921b583-7602-4b50-964e-35188ef6e2f6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.233281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab1a29-d601-44f9-84fc-c11272c44299 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.269034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f543f51c-cd86-49e4-902c-47df8065d069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.269678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543f51c-cd86-49e4-902c-47df8065d069 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.269954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543f51c-cd86-49e4-902c-47df8065d069 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.270226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543f51c-cd86-49e4-902c-47df8065d069 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.280719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f543f51c-cd86-49e4-902c-47df8065d069 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f543f51c-cd86-49e4-902c-47df8065d069 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:43.285405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:43.286312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:43.287114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 94/186] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:43.288324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe0ac20-788c-4a53-af09-cef4eb63622f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.288324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe0ac20-788c-4a53-af09-cef4eb63622f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.288324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe0ac20-788c-4a53-af09-cef4eb63622f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.304884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ebe0ac20-788c-4a53-af09-cef4eb63622f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:43.306079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ebe0ac20-788c-4a53-af09-cef4eb63622f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:43.331700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9921b583-7602-4b50-964e-35188ef6e2f6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.358311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:43.366805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa3bb83-8acb-4442-8a0d-69fbf7b10da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.367376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa3bb83-8acb-4442-8a0d-69fbf7b10da1 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.367650 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa3bb83-8acb-4442-8a0d-69fbf7b10da1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.367919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa3bb83-8acb-4442-8a0d-69fbf7b10da1 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.369033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa3bb83-8acb-4442-8a0d-69fbf7b10da1 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:43.370508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa3bb83-8acb-4442-8a0d-69fbf7b10da1 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:43.371163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 95/187] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:43.372015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe0ac20-788c-4a53-af09-cef4eb63622f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.372326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 93/188] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:43.373012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa3bb83-8acb-4442-8a0d-69fbf7b10da1 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.406127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:43.481197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa3bb83-8acb-4442-8a0d-69fbf7b10da1 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.485966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe0ac20-788c-4a53-af09-cef4eb63622f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.514846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e761d02-9ecc-445f-a85a-1d5e64e530d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.515480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e761d02-9ecc-445f-a85a-1d5e64e530d2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.515770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e761d02-9ecc-445f-a85a-1d5e64e530d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.516027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e761d02-9ecc-445f-a85a-1d5e64e530d2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.520821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.521580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.522015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.522439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.523231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:43.523677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:43.525164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:43.526100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e761d02-9ecc-445f-a85a-1d5e64e530d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-6e761d02-9ecc-445f-a85a-1d5e64e530d2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:43.531843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:43.532864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 94/189] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:43.538364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e761d02-9ecc-445f-a85a-1d5e64e530d2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.541693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 96/190] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:43.543054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.593801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:43.641489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e761d02-9ecc-445f-a85a-1d5e64e530d2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.641936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:43.648430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f0a248-a0c8-4442-a6b3-bcfdf5c27384 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.676139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.676751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.677034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.677309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.678070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] RBAC: Adding query filter params (role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=5492b71bdd6c44b6b273375d9450b759, user.id=055c8932e5b14530ad56960260a63d7e) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:43.678382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:43.679896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:43.683164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.683796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.684077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.684346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.685008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:43.685308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:43.685782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 95/191] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/role_assignments?user.id=055c8932e5b14530ad56960260a63d7e&scope.project.id=5492b71bdd6c44b6b273375d9450b759&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:43.692065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.692157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:43.700072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 97/192] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:43.700823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.808444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a313f598-2be9-4c72-bfcf-54aaa8fcc08b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.808811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66589abf-bde5-44b5-8edc-2af05fce41ad None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:43.826635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:43.847006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.847645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.847905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.848180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.848759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:43.848901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] RBAC: Adding query filter params (role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=8419cf9f31bb4753946ad654d60e9e26, user.id=055c8932e5b14530ad56960260a63d7e) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:43.849218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:43.849386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:43.849714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:43.849988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:43.850650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:43.850748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:43.850965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:43.852607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:43.857030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 96/193] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/role_assignments?user.id=055c8932e5b14530ad56960260a63d7e&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:43.861740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.865627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 98/194] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:43.866626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:43.891949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:43.996391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e3c188-a9f1-4135-8f02-7c2b4a925cf7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:44.005486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2de2efe-b19a-4660-905c-2496b6a8f0b5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:44.034328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b43fc-a323-4c0f-8374-a15301366e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:44.034977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b43fc-a323-4c0f-8374-a15301366e0e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:44.035274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b43fc-a323-4c0f-8374-a15301366e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:44.035533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b43fc-a323-4c0f-8374-a15301366e0e None admin] PATH_INFO: `/v3/users/094984b20c8844d39960145acac670f3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:44.049831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b43fc-a323-4c0f-8374-a15301366e0e None admin] RBAC: Authorizing `identity:get_user(user_id=094984b20c8844d39960145acac670f3)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:44.051375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b43fc-a323-4c0f-8374-a15301366e0e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:44.052738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5389f30a-b6dc-4a41-a784-2377a0e952c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:44.053019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 97/195] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/users/094984b20c8844d39960145acac670f3 => generated 283 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:44.053382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389f30a-b6dc-4a41-a784-2377a0e952c2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:44.053749 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389f30a-b6dc-4a41-a784-2377a0e952c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:44.053844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b43fc-a323-4c0f-8374-a15301366e0e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:44.054088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5389f30a-b6dc-4a41-a784-2377a0e952c2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:44.055171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5389f30a-b6dc-4a41-a784-2377a0e952c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:44.056601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5389f30a-b6dc-4a41-a784-2377a0e952c2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:44.060039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 99/196] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:44.061456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5389f30a-b6dc-4a41-a784-2377a0e952c2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:44.100399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:44.155152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:44.160192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b43fc-a323-4c0f-8374-a15301366e0e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:44.170908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5389f30a-b6dc-4a41-a784-2377a0e952c2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:44.194136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d45a4e-ec94-4ce5-affe-887f634924e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:44.194761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d45a4e-ec94-4ce5-affe-887f634924e5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:44.195037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d45a4e-ec94-4ce5-affe-887f634924e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:44.195301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d45a4e-ec94-4ce5-affe-887f634924e5 None admin] PATH_INFO: `/v3/users/094984b20c8844d39960145acac670f3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:44.196408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d45a4e-ec94-4ce5-affe-887f634924e5 None admin] RBAC: Authorizing `identity:get_user(user_id=094984b20c8844d39960145acac670f3)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:44.197871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d45a4e-ec94-4ce5-affe-887f634924e5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:44.199456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 98/197] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:44 2022] GET /identity/v3/users/094984b20c8844d39960145acac670f3 => generated 283 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:44.200200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d45a4e-ec94-4ce5-affe-887f634924e5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:44.208789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea087f90-19d6-434f-a05f-827254b20e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:44.209421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea087f90-19d6-434f-a05f-827254b20e16 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:44.209738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea087f90-19d6-434f-a05f-827254b20e16 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:44.210009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea087f90-19d6-434f-a05f-827254b20e16 None admin] PATH_INFO: `/v3/users/7fe723ad02f64965987bfb9686f25da9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:44.211131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea087f90-19d6-434f-a05f-827254b20e16 None admin] RBAC: Authorizing `identity:get_user(user_id=7fe723ad02f64965987bfb9686f25da9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:44.212810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea087f90-19d6-434f-a05f-827254b20e16 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:44.218918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 100/198] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:44 2022] GET /identity/v3/users/7fe723ad02f64965987bfb9686f25da9 => generated 289 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:44.218918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea087f90-19d6-434f-a05f-827254b20e16 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:44.303889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d45a4e-ec94-4ce5-affe-887f634924e5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:44.312059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea087f90-19d6-434f-a05f-827254b20e16 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:44.312748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:44.336512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adff5336-1992-438d-8b8f-29dbc0c03e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:44.337379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adff5336-1992-438d-8b8f-29dbc0c03e85 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:44.337683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adff5336-1992-438d-8b8f-29dbc0c03e85 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:44.337949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adff5336-1992-438d-8b8f-29dbc0c03e85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:44.345857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39de8abb-bb01-46e0-b109-b5fde78cb964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:44.346455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de8abb-bb01-46e0-b109-b5fde78cb964 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:44.346736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de8abb-bb01-46e0-b109-b5fde78cb964 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:44.347006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de8abb-bb01-46e0-b109-b5fde78cb964 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:44.347641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39de8abb-bb01-46e0-b109-b5fde78cb964 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:44.349144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39de8abb-bb01-46e0-b109-b5fde78cb964 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:44.404682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adff5336-1992-438d-8b8f-29dbc0c03e85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:44.406245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adff5336-1992-438d-8b8f-29dbc0c03e85 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:44.470693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 101/199] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:52:44 2022] POST /identity/v3/users => generated 297 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:44.471376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39de8abb-bb01-46e0-b109-b5fde78cb964 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:44.485279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:44.518032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 99/200] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:44.518883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adff5336-1992-438d-8b8f-29dbc0c03e85 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:44.530302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:44.586570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39de8abb-bb01-46e0-b109-b5fde78cb964 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:44.629233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adff5336-1992-438d-8b8f-29dbc0c03e85 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:44.630449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d4fd1e-617e-4984-967f-e040b6375054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:44.631365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4fd1e-617e-4984-967f-e040b6375054 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:44.632572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4fd1e-617e-4984-967f-e040b6375054 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:44.632852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4fd1e-617e-4984-967f-e040b6375054 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:44.671986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b26d1e5-65b1-4e1c-abaf-8d86986c201d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:44.671986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26d1e5-65b1-4e1c-abaf-8d86986c201d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:44.672292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26d1e5-65b1-4e1c-abaf-8d86986c201d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:44.672517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26d1e5-65b1-4e1c-abaf-8d86986c201d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:44.714462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d4fd1e-617e-4984-967f-e040b6375054 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:44.715960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d4fd1e-617e-4984-967f-e040b6375054 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:44.728921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:44.762286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b26d1e5-65b1-4e1c-abaf-8d86986c201d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:44.763763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b26d1e5-65b1-4e1c-abaf-8d86986c201d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:44.777957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:44.856222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 102/201] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:44.857450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d4fd1e-617e-4984-967f-e040b6375054 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:44.878745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 100/202] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:44.879603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b26d1e5-65b1-4e1c-abaf-8d86986c201d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:44.994566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b26d1e5-65b1-4e1c-abaf-8d86986c201d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:45.004477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:45.007657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d4fd1e-617e-4984-967f-e040b6375054 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:45.022798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:45.031482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ba0a88-eb40-420f-846f-079277329ef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:45.035355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ba0a88-eb40-420f-846f-079277329ef5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:45.035355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ba0a88-eb40-420f-846f-079277329ef5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:45.035355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ba0a88-eb40-420f-846f-079277329ef5 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:45.035355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ba0a88-eb40-420f-846f-079277329ef5 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:45.036573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ba0a88-eb40-420f-846f-079277329ef5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:45.038818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 101/203] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:44 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:45.039635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ba0a88-eb40-420f-846f-079277329ef5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:45.053329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56804170-d8eb-452f-b78e-90de4d73f5f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:45.054236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56804170-d8eb-452f-b78e-90de4d73f5f9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:45.055035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56804170-d8eb-452f-b78e-90de4d73f5f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:45.055515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56804170-d8eb-452f-b78e-90de4d73f5f9 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:45.057440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56804170-d8eb-452f-b78e-90de4d73f5f9 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:45.059853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56804170-d8eb-452f-b78e-90de4d73f5f9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:45.062347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 103/204] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:44 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:45.064500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56804170-d8eb-452f-b78e-90de4d73f5f9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:45.168298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ba0a88-eb40-420f-846f-079277329ef5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:45.197831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56804170-d8eb-452f-b78e-90de4d73f5f9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:45.208020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b931d4-2a01-408a-98f5-56665c0ab0ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:45.208835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b931d4-2a01-408a-98f5-56665c0ab0ae None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:45.209376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b931d4-2a01-408a-98f5-56665c0ab0ae None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:45.209818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b931d4-2a01-408a-98f5-56665c0ab0ae None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:45.211090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b931d4-2a01-408a-98f5-56665c0ab0ae None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:45.213140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b931d4-2a01-408a-98f5-56665c0ab0ae None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:45.215875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 102/205] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:45 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:45.216936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b931d4-2a01-408a-98f5-56665c0ab0ae None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:45.248714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:45.250097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:45.250778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:45.251223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:45.252372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] RBAC: Adding query filter params (role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16, user.id=094984b20c8844d39960145acac670f3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:45.253224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:45.254870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:45.262644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 104/206] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:45 2022] GET /identity/v3/role_assignments?user.id=094984b20c8844d39960145acac670f3&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:45.264216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:45.300978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:45.360871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:45.414478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b931d4-2a01-408a-98f5-56665c0ab0ae None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:45.454782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fc2c06-d4e4-4001-9d0b-c51f3b331797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:45.455108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc2c06-d4e4-4001-9d0b-c51f3b331797 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:45.455353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc2c06-d4e4-4001-9d0b-c51f3b331797 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:45.455619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc2c06-d4e4-4001-9d0b-c51f3b331797 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:45.456337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fc2c06-d4e4-4001-9d0b-c51f3b331797 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16, user.id=094984b20c8844d39960145acac670f3) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:45.456641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fc2c06-d4e4-4001-9d0b-c51f3b331797 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:45.459348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fc2c06-d4e4-4001-9d0b-c51f3b331797 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:45.466091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 103/207] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:45 2022] GET /identity/v3/role_assignments?user.id=094984b20c8844d39960145acac670f3&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 259 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:45.479354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea43b39-a3e4-4be1-9d36-fa0f55d5eeab None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:45.529456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bcc53b-d43d-4475-93d9-1eebc59d266a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:45.530419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcc53b-d43d-4475-93d9-1eebc59d266a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:45.530921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcc53b-d43d-4475-93d9-1eebc59d266a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:45.531372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcc53b-d43d-4475-93d9-1eebc59d266a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:45.532297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bcc53b-d43d-4475-93d9-1eebc59d266a None admin] RBAC: Adding query filter params (role.id=4ae9fa85613b4396ba3b5e4c3eff1cff, scope.project.id=8419cf9f31bb4753946ad654d60e9e26, user.id=7fe723ad02f64965987bfb9686f25da9) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:45.532751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bcc53b-d43d-4475-93d9-1eebc59d266a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:45.534409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bcc53b-d43d-4475-93d9-1eebc59d266a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:45.541735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 105/208] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:45 2022] GET /identity/v3/role_assignments?user.id=7fe723ad02f64965987bfb9686f25da9&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=4ae9fa85613b4396ba3b5e4c3eff1cff => generated 608 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:46.616607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e978f-097e-4858-a221-d3d9ba3452dc None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:46.617115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e978f-097e-4858-a221-d3d9ba3452dc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:46.617115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e978f-097e-4858-a221-d3d9ba3452dc None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:46.618682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 104/209] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:46.626417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e572bb5-9ba4-4efe-a262-c5c45779afa7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:46.627222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e572bb5-9ba4-4efe-a262-c5c45779afa7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:46.627715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e572bb5-9ba4-4efe-a262-c5c45779afa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:46.633800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9921aa38-908f-4f66-b4c4-93ceaa91ea5c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:46.634079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9921aa38-908f-4f66-b4c4-93ceaa91ea5c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:46.634589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9921aa38-908f-4f66-b4c4-93ceaa91ea5c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:46.636077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 105/210] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:46.641141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3f4fb8-4ab0-4a11-85a4-9b6ace5ef082 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:46.641671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3f4fb8-4ab0-4a11-85a4-9b6ace5ef082 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:46.642144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3f4fb8-4ab0-4a11-85a4-9b6ace5ef082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:46.648976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3e572bb5-9ba4-4efe-a262-c5c45779afa7 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:46.650846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3e572bb5-9ba4-4efe-a262-c5c45779afa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:46.662571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6d3f4fb8-4ab0-4a11-85a4-9b6ace5ef082 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:46.664183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6d3f4fb8-4ab0-4a11-85a4-9b6ace5ef082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:46.723594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 106/211] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:46.741347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 106/212] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:46.989891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e572bb5-9ba4-4efe-a262-c5c45779afa7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:47.010867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3f4fb8-4ab0-4a11-85a4-9b6ace5ef082 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:47.026436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:47.055175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:47.094319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e572bb5-9ba4-4efe-a262-c5c45779afa7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:47.119455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3f4fb8-4ab0-4a11-85a4-9b6ace5ef082 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:47.131022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93ab017-c7f2-4492-9553-73d2c16ea0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:47.131875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93ab017-c7f2-4492-9553-73d2c16ea0c3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.132298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93ab017-c7f2-4492-9553-73d2c16ea0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.132747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93ab017-c7f2-4492-9553-73d2c16ea0c3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.144535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93ab017-c7f2-4492-9553-73d2c16ea0c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b93ab017-c7f2-4492-9553-73d2c16ea0c3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:47.149615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:47.150480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:47.151278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:47.153645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 107/213] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:47.158907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356d565-8b1e-4e9b-8e8f-275d6de50e09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:47.159550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356d565-8b1e-4e9b-8e8f-275d6de50e09 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.159825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356d565-8b1e-4e9b-8e8f-275d6de50e09 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.160003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93ab017-c7f2-4492-9553-73d2c16ea0c3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:47.160097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9356d565-8b1e-4e9b-8e8f-275d6de50e09 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.161190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9356d565-8b1e-4e9b-8e8f-275d6de50e09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:47.162677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9356d565-8b1e-4e9b-8e8f-275d6de50e09 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:47.169356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 107/214] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:47.170365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356d565-8b1e-4e9b-8e8f-275d6de50e09 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:47.264711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:47.270985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93ab017-c7f2-4492-9553-73d2c16ea0c3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:47.280252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9356d565-8b1e-4e9b-8e8f-275d6de50e09 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:47.287893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:47.308002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:47.308960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.309457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.310029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.310905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:47.311402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:47.312964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:47.317493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fbad06-6b49-4415-bec6-d45d013e4e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:47.318264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fbad06-6b49-4415-bec6-d45d013e4e43 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.318584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fbad06-6b49-4415-bec6-d45d013e4e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.318860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fbad06-6b49-4415-bec6-d45d013e4e43 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.319500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fbad06-6b49-4415-bec6-d45d013e4e43 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:47.321028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fbad06-6b49-4415-bec6-d45d013e4e43 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:47.324620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 108/215] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:47.330254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:47.432973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfd4d3a-20c6-4d18-8812-69df06aa3cd2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:47.465983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 108/216] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:52:47 2022] POST /identity/v3/users => generated 297 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:47.473849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8aa886e-5228-475f-81e8-227f381f4b85 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.473849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8aa886e-5228-475f-81e8-227f381f4b85 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.473849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8aa886e-5228-475f-81e8-227f381f4b85 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.473849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 109/217] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:47.476372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea2b73-f6db-412d-9a63-81c9b4299180 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.476663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea2b73-f6db-412d-9a63-81c9b4299180 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.477324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea2b73-f6db-412d-9a63-81c9b4299180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.478862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a1fec2-a148-4a15-9e97-144db45ebe66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:47.479517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1fec2-a148-4a15-9e97-144db45ebe66 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.481916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1fec2-a148-4a15-9e97-144db45ebe66 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.482267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1fec2-a148-4a15-9e97-144db45ebe66 None admin] PATH_INFO: `/v3/users/da2c6b0b94d4439e83cf89cc4878fefc` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.500111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a1fec2-a148-4a15-9e97-144db45ebe66 None admin] RBAC: Authorizing `identity:get_user(user_id=da2c6b0b94d4439e83cf89cc4878fefc)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:47.501741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a1fec2-a148-4a15-9e97-144db45ebe66 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:47.508204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 109/218] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:47 2022] GET /identity/v3/users/da2c6b0b94d4439e83cf89cc4878fefc => generated 297 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:47.508204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9804efc-6558-4845-bd65-61749bc4c0b3 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.508204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9804efc-6558-4845-bd65-61749bc4c0b3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.508528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-98ea2b73-f6db-412d-9a63-81c9b4299180 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:47.513342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9804efc-6558-4845-bd65-61749bc4c0b3 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.513468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-98ea2b73-f6db-412d-9a63-81c9b4299180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:47.513560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 110/219] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:47.513560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9804efc-6558-4845-bd65-61749bc4c0b3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:47.531268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:47.591563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 110/220] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:47.592971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9f6616-05a4-4ef0-8e90-3178a0a12eca None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.593298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9f6616-05a4-4ef0-8e90-3178a0a12eca None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.593729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9f6616-05a4-4ef0-8e90-3178a0a12eca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.614737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4a9f6616-05a4-4ef0-8e90-3178a0a12eca None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:47.616226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4a9f6616-05a4-4ef0-8e90-3178a0a12eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:47.640922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9804efc-6558-4845-bd65-61749bc4c0b3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:47.687354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3852f66-3c64-4a05-8f06-541c13472620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:47.687981 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3852f66-3c64-4a05-8f06-541c13472620 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:47.688260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3852f66-3c64-4a05-8f06-541c13472620 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:47.688885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3852f66-3c64-4a05-8f06-541c13472620 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:47.710364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 111/221] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:47.764448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3852f66-3c64-4a05-8f06-541c13472620 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:47.765938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3852f66-3c64-4a05-8f06-541c13472620 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:47.787067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:47.869473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9f6616-05a4-4ef0-8e90-3178a0a12eca None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:47.874633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 111/222] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:47.880784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3852f66-3c64-4a05-8f06-541c13472620 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:47.899698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:47.994496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9f6616-05a4-4ef0-8e90-3178a0a12eca None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.019403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3852f66-3c64-4a05-8f06-541c13472620 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.037567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:48.044691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94d447-4bd7-4651-ac01-7184310544d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.045705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94d447-4bd7-4651-ac01-7184310544d9 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.050602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94d447-4bd7-4651-ac01-7184310544d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.050602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94d447-4bd7-4651-ac01-7184310544d9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.055787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22706a25-bc2b-4f39-8475-4abfa421d9e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.056498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22706a25-bc2b-4f39-8475-4abfa421d9e3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.056784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22706a25-bc2b-4f39-8475-4abfa421d9e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.057054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22706a25-bc2b-4f39-8475-4abfa421d9e3 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.058345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22706a25-bc2b-4f39-8475-4abfa421d9e3 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:48.059809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22706a25-bc2b-4f39-8475-4abfa421d9e3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:48.060014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af94d447-4bd7-4651-ac01-7184310544d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:48.061799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 112/223] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:47 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:48.062665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22706a25-bc2b-4f39-8475-4abfa421d9e3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-af94d447-4bd7-4651-ac01-7184310544d9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:48.067047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:48.067943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:48.069130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 112/224] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:52:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:48.069836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94d447-4bd7-4651-ac01-7184310544d9 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:48.178892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22706a25-bc2b-4f39-8475-4abfa421d9e3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.187947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:48.212601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94d447-4bd7-4651-ac01-7184310544d9 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.215723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2065d7-3407-4bd6-8ab9-b7d664c9b09c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.216444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2065d7-3407-4bd6-8ab9-b7d664c9b09c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.216779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2065d7-3407-4bd6-8ab9-b7d664c9b09c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.217644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2065d7-3407-4bd6-8ab9-b7d664c9b09c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.242499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2065d7-3407-4bd6-8ab9-b7d664c9b09c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-cb2065d7-3407-4bd6-8ab9-b7d664c9b09c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:48.251113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:48.251985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:48.252789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 113/225] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:48.258126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2065d7-3407-4bd6-8ab9-b7d664c9b09c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:48.263710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.264509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.264895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.265258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.266120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] RBAC: Adding query filter params (role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16, user.id=da2c6b0b94d4439e83cf89cc4878fefc) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:48.266523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:48.268182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:48.274522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 113/226] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:48 2022] GET /identity/v3/role_assignments?user.id=da2c6b0b94d4439e83cf89cc4878fefc&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 259 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:48.280825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:48.318333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:48.384509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2065d7-3407-4bd6-8ab9-b7d664c9b09c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.426238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.426861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.427348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.427656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.428756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:48.429070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:48.430518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:48.441746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 114/227] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:52:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:48.455547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:48.513162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71109936-591d-4a46-8e2d-6d3cabd61b5e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.551363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.551981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.552267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.552543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.553208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:48.553518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:48.554938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:48.564943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 114/228] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:48.570454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:48.584784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:48.684231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe51831a-aabc-4cf9-87ed-cfb0ec18a1ac None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.721498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d2972-be45-47d6-9480-9186bb8dd466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.722264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d2972-be45-47d6-9480-9186bb8dd466 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.722590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d2972-be45-47d6-9480-9186bb8dd466 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.722870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2d2972-be45-47d6-9480-9186bb8dd466 None admin] PATH_INFO: `/v3/users/094984b20c8844d39960145acac670f3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.724021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2d2972-be45-47d6-9480-9186bb8dd466 None admin] RBAC: Authorizing `identity:get_user(user_id=094984b20c8844d39960145acac670f3)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:48.725477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2d2972-be45-47d6-9480-9186bb8dd466 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:48.727533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 115/229] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:48 2022] GET /identity/v3/users/094984b20c8844d39960145acac670f3 => generated 283 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:48.729621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc6a876-b265-4ff7-a55f-4b8ce1731e69 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.734911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d2972-be45-47d6-9480-9186bb8dd466 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:48.742785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:48.802372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcc07af-bdc4-4ec6-9c3a-59d8593cfc79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.803013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc07af-bdc4-4ec6-9c3a-59d8593cfc79 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.803294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc07af-bdc4-4ec6-9c3a-59d8593cfc79 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.803566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcc07af-bdc4-4ec6-9c3a-59d8593cfc79 None admin] PATH_INFO: `/v3/users/094984b20c8844d39960145acac670f3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.804972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcc07af-bdc4-4ec6-9c3a-59d8593cfc79 None admin] RBAC: Authorizing `identity:get_user(user_id=094984b20c8844d39960145acac670f3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:48.808579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcc07af-bdc4-4ec6-9c3a-59d8593cfc79 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:48.810329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 115/230] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:48 2022] GET /identity/v3/users/094984b20c8844d39960145acac670f3 => generated 283 bytes in 355 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:48.815799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcc07af-bdc4-4ec6-9c3a-59d8593cfc79 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:48.871497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:48.914303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2d2972-be45-47d6-9480-9186bb8dd466 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:48.956397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61178b19-a4f3-4329-af6d-53f5a95d5cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:48.957043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61178b19-a4f3-4329-af6d-53f5a95d5cb2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:48.957324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61178b19-a4f3-4329-af6d-53f5a95d5cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:48.957672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61178b19-a4f3-4329-af6d-53f5a95d5cb2 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:48.959162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61178b19-a4f3-4329-af6d-53f5a95d5cb2 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:48.960785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61178b19-a4f3-4329-af6d-53f5a95d5cb2 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:48.963168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 116/231] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:48 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:48.968936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61178b19-a4f3-4329-af6d-53f5a95d5cb2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:49.027713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcc07af-bdc4-4ec6-9c3a-59d8593cfc79 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:49.087262 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61178b19-a4f3-4329-af6d-53f5a95d5cb2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:49.114349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6717bf-02f7-4a1e-912d-2decac1aa946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:49.115199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6717bf-02f7-4a1e-912d-2decac1aa946 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:49.115674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6717bf-02f7-4a1e-912d-2decac1aa946 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:49.116082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6717bf-02f7-4a1e-912d-2decac1aa946 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:49.122416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6717bf-02f7-4a1e-912d-2decac1aa946 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:49.124241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6717bf-02f7-4a1e-912d-2decac1aa946 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:49.125376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121ef404-b686-4cab-89f1-ab7e19b43de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:49.126199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 116/232] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:48 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:49.128487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ef404-b686-4cab-89f1-ab7e19b43de3 None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:49.128487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ef404-b686-4cab-89f1-ab7e19b43de3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:49.128487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121ef404-b686-4cab-89f1-ab7e19b43de3 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/users/094984b20c8844d39960145acac670f3/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:49.129225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121ef404-b686-4cab-89f1-ab7e19b43de3 None admin] RBAC: Authorizing `identity:create_grant(project_id=2c8e2a16d7d34db6886ebfb9d8689b16, user_id=094984b20c8844d39960145acac670f3, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:49.130939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121ef404-b686-4cab-89f1-ab7e19b43de3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:49.132305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6717bf-02f7-4a1e-912d-2decac1aa946 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:49.185447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:49.224561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 117/233] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:48 2022] PUT /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/users/094984b20c8844d39960145acac670f3/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 04:52:49.262027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6717bf-02f7-4a1e-912d-2decac1aa946 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:49.305153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0be13a6-f429-455d-9141-fb15b857c442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:49.305898 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be13a6-f429-455d-9141-fb15b857c442 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:49.306181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be13a6-f429-455d-9141-fb15b857c442 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:49.306451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0be13a6-f429-455d-9141-fb15b857c442 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/users/094984b20c8844d39960145acac670f3/roles/609ec30d147941b4a2b5508980aea7d0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:49.310792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0be13a6-f429-455d-9141-fb15b857c442 None admin] RBAC: Authorizing `identity:create_grant(project_id=2c8e2a16d7d34db6886ebfb9d8689b16, user_id=094984b20c8844d39960145acac670f3, role_id=609ec30d147941b4a2b5508980aea7d0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:49.312304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0be13a6-f429-455d-9141-fb15b857c442 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:49.448358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 117/234] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:49 2022] PUT /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/users/094984b20c8844d39960145acac670f3/roles/609ec30d147941b4a2b5508980aea7d0 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:49.872238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28274f95-037e-4eba-9798-192f045e69b5 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:49.872717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28274f95-037e-4eba-9798-192f045e69b5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:49.872790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28274f95-037e-4eba-9798-192f045e69b5 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:49.874421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 118/235] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:49.875916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9f340-1865-47cd-add1-5d0691c1cd55 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:49.876291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9f340-1865-47cd-add1-5d0691c1cd55 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:49.876466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9f340-1865-47cd-add1-5d0691c1cd55 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:49.878097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 118/236] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:49.883524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f839a-e556-4724-9727-3d686181fd54 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:49.883810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f839a-e556-4724-9727-3d686181fd54 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:49.884085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f839a-e556-4724-9727-3d686181fd54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:49.892923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:49.897292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecad8c1-4e11-4a03-9676-4df4ee298cea None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:49.897580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecad8c1-4e11-4a03-9676-4df4ee298cea None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:49.897891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecad8c1-4e11-4a03-9676-4df4ee298cea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:49.912732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-eb2f839a-e556-4724-9727-3d686181fd54 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:49.913957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-eb2f839a-e556-4724-9727-3d686181fd54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:49.926603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1ecad8c1-4e11-4a03-9676-4df4ee298cea None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:49.927789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1ecad8c1-4e11-4a03-9676-4df4ee298cea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:50.000629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 119/237] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:52:50.023849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 119/238] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:50.389346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f839a-e556-4724-9727-3d686181fd54 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:50.409036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecad8c1-4e11-4a03-9676-4df4ee298cea None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:50.460406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:50.525183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f839a-e556-4724-9727-3d686181fd54 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:50.563014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecad8c1-4e11-4a03-9676-4df4ee298cea None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:50.600824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dc1847-5a34-4340-b4d6-337ca0a2bdb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:50.601471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc1847-5a34-4340-b4d6-337ca0a2bdb5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:50.601823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc1847-5a34-4340-b4d6-337ca0a2bdb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:50.602173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc1847-5a34-4340-b4d6-337ca0a2bdb5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:50.607552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a57754-69c9-40bd-93b2-c136e47bd4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:50.608445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a57754-69c9-40bd-93b2-c136e47bd4f3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:50.608916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a57754-69c9-40bd-93b2-c136e47bd4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:50.609374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a57754-69c9-40bd-93b2-c136e47bd4f3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:50.613638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dc1847-5a34-4340-b4d6-337ca0a2bdb5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:50.614480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a57754-69c9-40bd-93b2-c136e47bd4f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:50.616149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a57754-69c9-40bd-93b2-c136e47bd4f3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:50.618256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 120/239] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-27dc1847-5a34-4340-b4d6-337ca0a2bdb5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:50.618694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:50.619551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:50.620344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 120/240] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:50.623010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88712d3-29ab-4def-84c1-37362ffd282c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:50.623499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88712d3-29ab-4def-84c1-37362ffd282c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:50.623958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88712d3-29ab-4def-84c1-37362ffd282c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:50.625369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 121/241] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:50.626277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88712d3-29ab-4def-84c1-37362ffd282c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:50.626379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dc1847-5a34-4340-b4d6-337ca0a2bdb5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:50.629303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:50.733624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:50.743758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88712d3-29ab-4def-84c1-37362ffd282c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:50.748312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dc1847-5a34-4340-b4d6-337ca0a2bdb5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:50.786042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ff14a5-11a0-4fd3-a6a5-92b5bb63c2eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:50.786925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ff14a5-11a0-4fd3-a6a5-92b5bb63c2eb None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:50.787273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ff14a5-11a0-4fd3-a6a5-92b5bb63c2eb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:50.787636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ff14a5-11a0-4fd3-a6a5-92b5bb63c2eb None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:50.788496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ff14a5-11a0-4fd3-a6a5-92b5bb63c2eb None admin] RBAC: Adding query filter params (name=reader) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:50.788800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ff14a5-11a0-4fd3-a6a5-92b5bb63c2eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:50.789548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8933e074-5b2f-4fe1-a5d9-0e51d1623728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:50.790289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ff14a5-11a0-4fd3-a6a5-92b5bb63c2eb None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:50.790401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8933e074-5b2f-4fe1-a5d9-0e51d1623728 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:50.790796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8933e074-5b2f-4fe1-a5d9-0e51d1623728 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:50.791138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8933e074-5b2f-4fe1-a5d9-0e51d1623728 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:50.791963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8933e074-5b2f-4fe1-a5d9-0e51d1623728 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:50.794171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8933e074-5b2f-4fe1-a5d9-0e51d1623728 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:50.799165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 122/242] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:50.800341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe706cb-b5a2-4f82-967e-59a176fff5a4 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:50.800622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe706cb-b5a2-4f82-967e-59a176fff5a4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:50.800892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe706cb-b5a2-4f82-967e-59a176fff5a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:50.819029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ffe706cb-b5a2-4f82-967e-59a176fff5a4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:50.820210 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ffe706cb-b5a2-4f82-967e-59a176fff5a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:50.881458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:50.890683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 123/243] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:50.891639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe706cb-b5a2-4f82-967e-59a176fff5a4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:50.901438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 121/244] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:52:50 2022] POST /identity/v3/users => generated 293 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:51.052813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe706cb-b5a2-4f82-967e-59a176fff5a4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:51.098085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16439fd6-2d3f-44ae-a26b-924abda61b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:51.098729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16439fd6-2d3f-44ae-a26b-924abda61b3d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.099008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16439fd6-2d3f-44ae-a26b-924abda61b3d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.099280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16439fd6-2d3f-44ae-a26b-924abda61b3d None admin] PATH_INFO: `/v3/users/2e8c1ee74cba40f794834d08976d59d4` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.113465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16439fd6-2d3f-44ae-a26b-924abda61b3d None admin] RBAC: Authorizing `identity:get_user(user_id=2e8c1ee74cba40f794834d08976d59d4)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:51.114969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16439fd6-2d3f-44ae-a26b-924abda61b3d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:51.116787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 124/245] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:50 2022] GET /identity/v3/users/2e8c1ee74cba40f794834d08976d59d4 => generated 297 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:51.122898 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8933e074-5b2f-4fe1-a5d9-0e51d1623728 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:51.233480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8933e074-5b2f-4fe1-a5d9-0e51d1623728 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:51.271407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5748ab-7550-4637-96d0-341875720897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:51.272033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5748ab-7550-4637-96d0-341875720897 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.272311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5748ab-7550-4637-96d0-341875720897 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.272585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5748ab-7550-4637-96d0-341875720897 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.291761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:51.345911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5748ab-7550-4637-96d0-341875720897 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:51.347315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5748ab-7550-4637-96d0-341875720897 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:51.368188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16439fd6-2d3f-44ae-a26b-924abda61b3d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:51.439416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:51.455783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 122/246] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:51.461870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5748ab-7550-4637-96d0-341875720897 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:51.494411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16439fd6-2d3f-44ae-a26b-924abda61b3d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:51.514734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:51.533618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157d1bff-689c-4c3e-aebe-52c7744623f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:51.534620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d1bff-689c-4c3e-aebe-52c7744623f5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.535089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d1bff-689c-4c3e-aebe-52c7744623f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.535544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d1bff-689c-4c3e-aebe-52c7744623f5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.547511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157d1bff-689c-4c3e-aebe-52c7744623f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-157d1bff-689c-4c3e-aebe-52c7744623f5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:51.554462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:51.555439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:51.556232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:51.558293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 125/247] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:52:51.560804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214a950f-0abf-430e-aeae-7449aa4da6ee None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.560804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214a950f-0abf-430e-aeae-7449aa4da6ee None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.560804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214a950f-0abf-430e-aeae-7449aa4da6ee None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.565673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 126/248] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:51.568555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214a950f-0abf-430e-aeae-7449aa4da6ee None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:51.578657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5748ab-7550-4637-96d0-341875720897 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:51.621101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe7e607-87a1-4ac2-89b8-14df58ffd0fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:51.622148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e607-87a1-4ac2-89b8-14df58ffd0fa None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.622428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e607-87a1-4ac2-89b8-14df58ffd0fa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.622704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e607-87a1-4ac2-89b8-14df58ffd0fa None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.624525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe7e607-87a1-4ac2-89b8-14df58ffd0fa None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:51.626031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe7e607-87a1-4ac2-89b8-14df58ffd0fa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:51.627848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 123/249] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:51 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:51.628804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d803-4d5d-41c1-86b4-20bcc3af320d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.629081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d803-4d5d-41c1-86b4-20bcc3af320d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.629351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315d803-4d5d-41c1-86b4-20bcc3af320d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.649905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2315d803-4d5d-41c1-86b4-20bcc3af320d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:51.651081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2315d803-4d5d-41c1-86b4-20bcc3af320d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:51.703592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214a950f-0abf-430e-aeae-7449aa4da6ee None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:51.722846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:51.741705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 124/250] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:51.742643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315d803-4d5d-41c1-86b4-20bcc3af320d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:51.746567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:51.747188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.747462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.747817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.748479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:51.748782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:51.750308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:51.763203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 127/251] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:51.766823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:51.810284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:51.880787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d266667b-9430-4ba8-b728-56d8580e0a91 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:51.925943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b44839f-4bfc-4131-a816-ad0e52b97e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:51.926560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b44839f-4bfc-4131-a816-ad0e52b97e06 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.926845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b44839f-4bfc-4131-a816-ad0e52b97e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.927113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b44839f-4bfc-4131-a816-ad0e52b97e06 None admin] PATH_INFO: `/v3/users/da2c6b0b94d4439e83cf89cc4878fefc` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.930342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b44839f-4bfc-4131-a816-ad0e52b97e06 None admin] RBAC: Authorizing `identity:get_user(user_id=da2c6b0b94d4439e83cf89cc4878fefc)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:51.932044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b44839f-4bfc-4131-a816-ad0e52b97e06 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:51.934792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 128/252] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:51 2022] GET /identity/v3/users/da2c6b0b94d4439e83cf89cc4878fefc => generated 297 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:51.936141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098a9ef5-9a13-4638-97d3-c8276c3bb348 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.936622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098a9ef5-9a13-4638-97d3-c8276c3bb348 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.937167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098a9ef5-9a13-4638-97d3-c8276c3bb348 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.942884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315d803-4d5d-41c1-86b4-20bcc3af320d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:51.944045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 129/253] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:51 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:51.944984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098a9ef5-9a13-4638-97d3-c8276c3bb348 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:51.980579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030812ad-8328-460f-9c24-c0ed731278d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:51.981245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030812ad-8328-460f-9c24-c0ed731278d0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.981573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030812ad-8328-460f-9c24-c0ed731278d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.981872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030812ad-8328-460f-9c24-c0ed731278d0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:51.982642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030812ad-8328-460f-9c24-c0ed731278d0 None admin] RBAC: Adding query filter params (role.id=4ae9fa85613b4396ba3b5e4c3eff1cff, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16, user.id=2e8c1ee74cba40f794834d08976d59d4) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:51.982950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030812ad-8328-460f-9c24-c0ed731278d0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:51.989613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030812ad-8328-460f-9c24-c0ed731278d0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:51.990630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 125/254] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:51 2022] GET /identity/v3/role_assignments?user.id=2e8c1ee74cba40f794834d08976d59d4&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=4ae9fa85613b4396ba3b5e4c3eff1cff => generated 259 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:51.991471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b04abe-cd6e-4da7-a26b-1614f4510295 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:51.991756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b04abe-cd6e-4da7-a26b-1614f4510295 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:51.992024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b04abe-cd6e-4da7-a26b-1614f4510295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:52.011262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f1b04abe-cd6e-4da7-a26b-1614f4510295 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:52.017617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f1b04abe-cd6e-4da7-a26b-1614f4510295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:52.033763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:52.117944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 126/255] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:52.118905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b04abe-cd6e-4da7-a26b-1614f4510295 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:52.119215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098a9ef5-9a13-4638-97d3-c8276c3bb348 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:52.157197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e220b0b-7848-4437-b652-735be466a5b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:52.158236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e220b0b-7848-4437-b652-735be466a5b8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:52.158725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e220b0b-7848-4437-b652-735be466a5b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:52.159197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e220b0b-7848-4437-b652-735be466a5b8 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:52.160642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e220b0b-7848-4437-b652-735be466a5b8 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:52.162427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e220b0b-7848-4437-b652-735be466a5b8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:52.164404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 130/256] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:51 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:52.170809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e220b0b-7848-4437-b652-735be466a5b8 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:52.206122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:52.342909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e220b0b-7848-4437-b652-735be466a5b8 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:52.385963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:52.395951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b89187f-8a34-4d00-82a9-514d9eb3f4e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:52.396637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b89187f-8a34-4d00-82a9-514d9eb3f4e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:52.396921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b89187f-8a34-4d00-82a9-514d9eb3f4e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:52.397234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b89187f-8a34-4d00-82a9-514d9eb3f4e0 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/users/da2c6b0b94d4439e83cf89cc4878fefc/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:52.399520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b89187f-8a34-4d00-82a9-514d9eb3f4e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=2c8e2a16d7d34db6886ebfb9d8689b16, user_id=da2c6b0b94d4439e83cf89cc4878fefc, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:52.401021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b89187f-8a34-4d00-82a9-514d9eb3f4e0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:52.426828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b04abe-cd6e-4da7-a26b-1614f4510295 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:52.460679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dfbe6e-d473-4e62-b953-9714a8f3f0cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:52.461434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfbe6e-d473-4e62-b953-9714a8f3f0cf None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:52.461740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfbe6e-d473-4e62-b953-9714a8f3f0cf None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:52.462317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfbe6e-d473-4e62-b953-9714a8f3f0cf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:52.473810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dfbe6e-d473-4e62-b953-9714a8f3f0cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-a1dfbe6e-d473-4e62-b953-9714a8f3f0cf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:52.478817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:52.479707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:52.480942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 127/257] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:52:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:52.481625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dfbe6e-d473-4e62-b953-9714a8f3f0cf None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:52.529273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 131/258] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:52 2022] PUT /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/users/da2c6b0b94d4439e83cf89cc4878fefc/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:52.531042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b89187f-8a34-4d00-82a9-514d9eb3f4e0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:52.632544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:52.638574 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dfbe6e-d473-4e62-b953-9714a8f3f0cf None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:52.713695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d519c43-be31-46a9-8127-0fd7a357c9e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:52.713695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d519c43-be31-46a9-8127-0fd7a357c9e6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:52.713695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d519c43-be31-46a9-8127-0fd7a357c9e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:52.713695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d519c43-be31-46a9-8127-0fd7a357c9e6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:52.741216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d519c43-be31-46a9-8127-0fd7a357c9e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0d519c43-be31-46a9-8127-0fd7a357c9e6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:52.747825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:52.748730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:52.750180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 128/259] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:52:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:52.754837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d519c43-be31-46a9-8127-0fd7a357c9e6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:52.797501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b89187f-8a34-4d00-82a9-514d9eb3f4e0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:52.874934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:52.875913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:52.876391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:52.876862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:52.877775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:52.878291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:52.879891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:52.881916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d519c43-be31-46a9-8127-0fd7a357c9e6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:52.893922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 132/260] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:52.899849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:52.920938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:52.921553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:52.921860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:52.922135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:52.922808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:52.923112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:52.924506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:52.933826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 129/261] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:52:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:52.939200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:52.965401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:52.969104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:53.045773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b1a483-98d5-4143-b587-63f1e75959c7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:53.105790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d9d38d-4e77-41ca-b196-4e3849f10f93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:53.106450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d9d38d-4e77-41ca-b196-4e3849f10f93 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:53.106732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d9d38d-4e77-41ca-b196-4e3849f10f93 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:53.107008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d9d38d-4e77-41ca-b196-4e3849f10f93 None admin] PATH_INFO: `/v3/users/094984b20c8844d39960145acac670f3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:53.108169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d9d38d-4e77-41ca-b196-4e3849f10f93 None admin] RBAC: Authorizing `identity:get_user(user_id=094984b20c8844d39960145acac670f3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:53.109652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d9d38d-4e77-41ca-b196-4e3849f10f93 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:53.111252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 130/262] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:52 2022] GET /identity/v3/users/094984b20c8844d39960145acac670f3 => generated 283 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:53.116676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d9d38d-4e77-41ca-b196-4e3849f10f93 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:53.162645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00a24e7-0876-4a77-9ed9-baf42c495f07 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:53.270891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357379c7-d99b-467f-9c62-0e4f8a5f8d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:53.271556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357379c7-d99b-467f-9c62-0e4f8a5f8d87 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:53.271837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357379c7-d99b-467f-9c62-0e4f8a5f8d87 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:53.272108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357379c7-d99b-467f-9c62-0e4f8a5f8d87 None admin] PATH_INFO: `/v3/users/094984b20c8844d39960145acac670f3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:53.273261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357379c7-d99b-467f-9c62-0e4f8a5f8d87 None admin] RBAC: Authorizing `identity:get_user(user_id=094984b20c8844d39960145acac670f3)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:53.274777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357379c7-d99b-467f-9c62-0e4f8a5f8d87 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:53.276786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 133/263] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:52 2022] GET /identity/v3/users/094984b20c8844d39960145acac670f3 => generated 283 bytes in 377 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:53.285054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d9d38d-4e77-41ca-b196-4e3849f10f93 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:53.286564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357379c7-d99b-467f-9c62-0e4f8a5f8d87 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:53.288575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:53.347465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebdd7d-5a0e-4cee-a005-e50694f4cda8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:53.348125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebdd7d-5a0e-4cee-a005-e50694f4cda8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:53.348407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebdd7d-5a0e-4cee-a005-e50694f4cda8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:53.348681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ebdd7d-5a0e-4cee-a005-e50694f4cda8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:53.487700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ebdd7d-5a0e-4cee-a005-e50694f4cda8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:53.489098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ebdd7d-5a0e-4cee-a005-e50694f4cda8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:53.582642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:53.586315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357379c7-d99b-467f-9c62-0e4f8a5f8d87 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:53.670837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8e070b-1ad4-4db8-a718-b6dfe1d3cacd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:53.671462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e070b-1ad4-4db8-a718-b6dfe1d3cacd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:53.671746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e070b-1ad4-4db8-a718-b6dfe1d3cacd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:53.672022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e070b-1ad4-4db8-a718-b6dfe1d3cacd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:53.705948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:53.736888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 131/264] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 620 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 25 04:52:53.742554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebdd7d-5a0e-4cee-a005-e50694f4cda8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:53.854686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e070b-1ad4-4db8-a718-b6dfe1d3cacd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:53.856688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e070b-1ad4-4db8-a718-b6dfe1d3cacd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:53.977340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ebdd7d-5a0e-4cee-a005-e50694f4cda8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:54.059384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3713eac-df92-4565-b5c3-377b3cc20715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:54.059952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3713eac-df92-4565-b5c3-377b3cc20715 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.060227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3713eac-df92-4565-b5c3-377b3cc20715 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.060494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3713eac-df92-4565-b5c3-377b3cc20715 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.066281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3713eac-df92-4565-b5c3-377b3cc20715 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:54.067735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3713eac-df92-4565-b5c3-377b3cc20715 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:54.069378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 132/265] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:53 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:54.075947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3713eac-df92-4565-b5c3-377b3cc20715 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:54.085019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:54.094897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 134/266] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 809 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:54.100341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8e070b-1ad4-4db8-a718-b6dfe1d3cacd None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:54.224996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:54.289660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8e070b-1ad4-4db8-a718-b6dfe1d3cacd None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:54.298851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3713eac-df92-4565-b5c3-377b3cc20715 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:54.336191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8f9ba4-9a6b-4dfe-9bdd-acfb1e0ba2d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:54.337290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8f9ba4-9a6b-4dfe-9bdd-acfb1e0ba2d9 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.337551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8f9ba4-9a6b-4dfe-9bdd-acfb1e0ba2d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.337867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8f9ba4-9a6b-4dfe-9bdd-acfb1e0ba2d9 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.339739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8f9ba4-9a6b-4dfe-9bdd-acfb1e0ba2d9 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:54.340464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b795b-2bb7-4026-a858-5ea9fb5efc36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:54.341067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b795b-2bb7-4026-a858-5ea9fb5efc36 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.341237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8f9ba4-9a6b-4dfe-9bdd-acfb1e0ba2d9 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:54.341327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b795b-2bb7-4026-a858-5ea9fb5efc36 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.341617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b795b-2bb7-4026-a858-5ea9fb5efc36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.342376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b795b-2bb7-4026-a858-5ea9fb5efc36 None admin] RBAC: Adding query filter params (role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16, user.id=094984b20c8844d39960145acac670f3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:54.342688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b795b-2bb7-4026-a858-5ea9fb5efc36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:54.343209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 135/267] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:54 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:54.344247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b795b-2bb7-4026-a858-5ea9fb5efc36 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:54.344425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65601102-d651-41ea-8e8a-eae876531d45 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.344686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65601102-d651-41ea-8e8a-eae876531d45 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.344966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65601102-d651-41ea-8e8a-eae876531d45 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.346770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 136/268] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:54.348213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c7c807-596f-439f-af32-11b57c636616 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.348498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c7c807-596f-439f-af32-11b57c636616 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.348774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c7c807-596f-439f-af32-11b57c636616 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.350632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 137/269] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:54.351339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c7c807-596f-439f-af32-11b57c636616 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:54.359921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 133/270] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:54 2022] GET /identity/v3/role_assignments?user.id=094984b20c8844d39960145acac670f3&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 608 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:54.361262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc8530-ed1f-4999-a126-c3d987a698fb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.361547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc8530-ed1f-4999-a126-c3d987a698fb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.361857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfc8530-ed1f-4999-a126-c3d987a698fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.408189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ecfc8530-ed1f-4999-a126-c3d987a698fb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:54.410933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ecfc8530-ed1f-4999-a126-c3d987a698fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:54.494140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:54.528785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 134/271] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:54.530098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8caf173f-cf0b-48d3-bf1b-cfdbbed9fad9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.530387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8caf173f-cf0b-48d3-bf1b-cfdbbed9fad9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.530696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8caf173f-cf0b-48d3-bf1b-cfdbbed9fad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.550745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8caf173f-cf0b-48d3-bf1b-cfdbbed9fad9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:54.551925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8caf173f-cf0b-48d3-bf1b-cfdbbed9fad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:54.561456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c7c807-596f-439f-af32-11b57c636616 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:54.599168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56df18f-b81e-40aa-a2ea-085ace78abbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:54.599848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56df18f-b81e-40aa-a2ea-085ace78abbe None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.600266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56df18f-b81e-40aa-a2ea-085ace78abbe None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.600538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56df18f-b81e-40aa-a2ea-085ace78abbe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.601306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56df18f-b81e-40aa-a2ea-085ace78abbe None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16, user.id=094984b20c8844d39960145acac670f3) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:54.602171 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56df18f-b81e-40aa-a2ea-085ace78abbe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:54.603750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56df18f-b81e-40aa-a2ea-085ace78abbe None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:54.606013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:54.610759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 138/272] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:54 2022] GET /identity/v3/role_assignments?user.id=094984b20c8844d39960145acac670f3&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 608 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:54.612569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960d0da7-511b-445a-af52-8133163ddf17 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.612926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960d0da7-511b-445a-af52-8133163ddf17 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.613269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960d0da7-511b-445a-af52-8133163ddf17 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.614570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 139/273] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:54.625106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a0b152-4895-4c31-a61f-8b789c428fc5 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:54.625390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a0b152-4895-4c31-a61f-8b789c428fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:54.625706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a0b152-4895-4c31-a61f-8b789c428fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:54.652662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 135/274] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:54.657972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f3a0b152-4895-4c31-a61f-8b789c428fc5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:54.659278 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f3a0b152-4895-4c31-a61f-8b789c428fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:54.739164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 140/275] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:54.952853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8caf173f-cf0b-48d3-bf1b-cfdbbed9fad9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.001314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a0b152-4895-4c31-a61f-8b789c428fc5 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.077825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:55.080654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8caf173f-cf0b-48d3-bf1b-cfdbbed9fad9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.118919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38aa8ebd-f6af-4b4e-a538-3a1cc94447af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.119844 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa8ebd-f6af-4b4e-a538-3a1cc94447af None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.120857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa8ebd-f6af-4b4e-a538-3a1cc94447af None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.121308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa8ebd-f6af-4b4e-a538-3a1cc94447af None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.122453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a0b152-4895-4c31-a61f-8b789c428fc5 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.132647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38aa8ebd-f6af-4b4e-a538-3a1cc94447af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-38aa8ebd-f6af-4b4e-a538-3a1cc94447af None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:55.138274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:55.139175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:55.139965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:55.146899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 136/276] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:55.147891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38aa8ebd-f6af-4b4e-a538-3a1cc94447af None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.157413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a79e27-e39c-4f7e-a3d7-8b2723158601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.158077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a79e27-e39c-4f7e-a3d7-8b2723158601 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.158353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a79e27-e39c-4f7e-a3d7-8b2723158601 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.158619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a79e27-e39c-4f7e-a3d7-8b2723158601 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.159705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a79e27-e39c-4f7e-a3d7-8b2723158601 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:55.161145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a79e27-e39c-4f7e-a3d7-8b2723158601 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:55.162716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 141/277] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:55.163455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a79e27-e39c-4f7e-a3d7-8b2723158601 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.193815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:55.250103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38aa8ebd-f6af-4b4e-a538-3a1cc94447af None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.265358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a79e27-e39c-4f7e-a3d7-8b2723158601 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.283708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90611380-6d98-4fa6-8482-08f72ebf1526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.284331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90611380-6d98-4fa6-8482-08f72ebf1526 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.284608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90611380-6d98-4fa6-8482-08f72ebf1526 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.284883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90611380-6d98-4fa6-8482-08f72ebf1526 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.293235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:55.295233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90611380-6d98-4fa6-8482-08f72ebf1526 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:55.298816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.299420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.299698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.299962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-90611380-6d98-4fa6-8482-08f72ebf1526 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:55.300047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:55.309650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:55.310456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:52:55.311082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:55.311082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:55.311205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 137/278] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:55.311205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e9a8c9-6370-4ad5-ad41-e4c1e4c02b3d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.311327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:55.311393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e9a8c9-6370-4ad5-ad41-e4c1e4c02b3d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.311393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e9a8c9-6370-4ad5-ad41-e4c1e4c02b3d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.311393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 138/279] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:55.311393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e9a8c9-6370-4ad5-ad41-e4c1e4c02b3d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.314233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 142/280] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:55.318522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.407139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e9a8c9-6370-4ad5-ad41-e4c1e4c02b3d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.408722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:55.415943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ba7387-e275-46d0-bfd2-6f92901bc629 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.440915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f71586f-b1ad-4d7a-8bcd-dbbad4f6c25c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.441580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f71586f-b1ad-4d7a-8bcd-dbbad4f6c25c None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.441989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f71586f-b1ad-4d7a-8bcd-dbbad4f6c25c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.442473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f71586f-b1ad-4d7a-8bcd-dbbad4f6c25c None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.443441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f71586f-b1ad-4d7a-8bcd-dbbad4f6c25c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:55.444879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f71586f-b1ad-4d7a-8bcd-dbbad4f6c25c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:55.450022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aebce3e-db35-49ce-b4ee-b027954836bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.450625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aebce3e-db35-49ce-b4ee-b027954836bc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.450904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aebce3e-db35-49ce-b4ee-b027954836bc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.451176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aebce3e-db35-49ce-b4ee-b027954836bc None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.451830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aebce3e-db35-49ce-b4ee-b027954836bc None admin] RBAC: Adding query filter params (name=reader) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:55.452132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aebce3e-db35-49ce-b4ee-b027954836bc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:55.453494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aebce3e-db35-49ce-b4ee-b027954836bc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:55.461666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 143/281] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:55.463063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e067d5e1-fa8b-43fa-b53b-6ab873ecf460 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.463345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e067d5e1-fa8b-43fa-b53b-6ab873ecf460 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.463616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e067d5e1-fa8b-43fa-b53b-6ab873ecf460 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.481065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e067d5e1-fa8b-43fa-b53b-6ab873ecf460 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:55.482157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e067d5e1-fa8b-43fa-b53b-6ab873ecf460 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:55.529174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:55.544448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 139/282] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:52:55 2022] POST /identity/v3/users => generated 293 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:55.545279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f71586f-b1ad-4d7a-8bcd-dbbad4f6c25c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.557162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 144/283] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:55.558614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e067d5e1-fa8b-43fa-b53b-6ab873ecf460 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.670918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e067d5e1-fa8b-43fa-b53b-6ab873ecf460 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.695490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f71586f-b1ad-4d7a-8bcd-dbbad4f6c25c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.710852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e48b63-269c-493b-8fc2-5eb4b766db0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.711557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e48b63-269c-493b-8fc2-5eb4b766db0b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.711838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e48b63-269c-493b-8fc2-5eb4b766db0b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.712366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e48b63-269c-493b-8fc2-5eb4b766db0b None admin] PATH_INFO: `/v3/users/2e8c1ee74cba40f794834d08976d59d4` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.713672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e48b63-269c-493b-8fc2-5eb4b766db0b None admin] RBAC: Authorizing `identity:get_user(user_id=2e8c1ee74cba40f794834d08976d59d4)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:55.715664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e48b63-269c-493b-8fc2-5eb4b766db0b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:55.717633 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 145/284] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/users/2e8c1ee74cba40f794834d08976d59d4 => generated 297 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:55.724421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e48b63-269c-493b-8fc2-5eb4b766db0b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.749457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623396ad-4f8a-4fdb-b516-58acec973b5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.750091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623396ad-4f8a-4fdb-b516-58acec973b5e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.750367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623396ad-4f8a-4fdb-b516-58acec973b5e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.750637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623396ad-4f8a-4fdb-b516-58acec973b5e None admin] PATH_INFO: `/v3/users/18e834b69a2844ff9d98a8dda3c3b110` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.767086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623396ad-4f8a-4fdb-b516-58acec973b5e None admin] RBAC: Authorizing `identity:get_user(user_id=18e834b69a2844ff9d98a8dda3c3b110)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:55.770325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623396ad-4f8a-4fdb-b516-58acec973b5e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:55.772907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 140/285] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/users/18e834b69a2844ff9d98a8dda3c3b110 => generated 293 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:55.779239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623396ad-4f8a-4fdb-b516-58acec973b5e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.795928 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:55.811705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:55.864957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e48b63-269c-493b-8fc2-5eb4b766db0b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.907384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2125fa-27f0-49ac-98b0-1507616115fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.908476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2125fa-27f0-49ac-98b0-1507616115fc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.909271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2125fa-27f0-49ac-98b0-1507616115fc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.909752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2125fa-27f0-49ac-98b0-1507616115fc None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.911097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2125fa-27f0-49ac-98b0-1507616115fc None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:55.912723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2125fa-27f0-49ac-98b0-1507616115fc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:55.914567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 146/286] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:55.917803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623396ad-4f8a-4fdb-b516-58acec973b5e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:55.920488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2125fa-27f0-49ac-98b0-1507616115fc None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:55.959401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:55.959756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:55.960006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:55.960279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:55.961188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] RBAC: Adding query filter params (role.id=0940d9778fb041fbaf7c4f4317feb2d5, user.id=18e834b69a2844ff9d98a8dda3c3b110, scope.system=all) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:55.962339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:55.964558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:55.973675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 141/287] 10.222.0.16 () {56 vars in 1308 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/role_assignments?user.id=18e834b69a2844ff9d98a8dda3c3b110&scope.system=all&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 226 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:55.974649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:56.103885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2125fa-27f0-49ac-98b0-1507616115fc None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:56.129091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:56.146139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:56.168112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208a77f2-382c-4203-94cf-bfc63ec9ebd9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:56.171866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213955db-a091-4070-bdd3-1bf34381e83d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:56.172786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213955db-a091-4070-bdd3-1bf34381e83d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:56.173259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213955db-a091-4070-bdd3-1bf34381e83d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:56.173750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213955db-a091-4070-bdd3-1bf34381e83d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:56.188484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213955db-a091-4070-bdd3-1bf34381e83d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-213955db-a091-4070-bdd3-1bf34381e83d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:56.195110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:56.196001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:56.196817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:56.201638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 147/288] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:56.211128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213955db-a091-4070-bdd3-1bf34381e83d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:56.223298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57696dd-e4cd-45f1-a84e-8e1260578aba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:56.224251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57696dd-e4cd-45f1-a84e-8e1260578aba None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:56.224715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57696dd-e4cd-45f1-a84e-8e1260578aba None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:56.225123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57696dd-e4cd-45f1-a84e-8e1260578aba None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/users/2e8c1ee74cba40f794834d08976d59d4/roles/4ae9fa85613b4396ba3b5e4c3eff1cff` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:56.230468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57696dd-e4cd-45f1-a84e-8e1260578aba None admin] RBAC: Authorizing `identity:create_grant(project_id=2c8e2a16d7d34db6886ebfb9d8689b16, user_id=2e8c1ee74cba40f794834d08976d59d4, role_id=4ae9fa85613b4396ba3b5e4c3eff1cff)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:56.234027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57696dd-e4cd-45f1-a84e-8e1260578aba None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:56.330175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213955db-a091-4070-bdd3-1bf34381e83d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:56.356952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 142/289] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:52:55 2022] PUT /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/users/2e8c1ee74cba40f794834d08976d59d4/roles/4ae9fa85613b4396ba3b5e4c3eff1cff => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:56.374584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:56.375230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:56.375520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:56.375796 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:56.376496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:56.376805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:56.378377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:56.388786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 148/290] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:56.394305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57696dd-e4cd-45f1-a84e-8e1260578aba None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:56.556723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57696dd-e4cd-45f1-a84e-8e1260578aba None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:56.593657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3683698b-deab-4e41-9563-412267d1100a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:56.594305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683698b-deab-4e41-9563-412267d1100a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:56.594584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683698b-deab-4e41-9563-412267d1100a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:56.594861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3683698b-deab-4e41-9563-412267d1100a None admin] PATH_INFO: `/v3/users/da2c6b0b94d4439e83cf89cc4878fefc` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:56.596004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3683698b-deab-4e41-9563-412267d1100a None admin] RBAC: Authorizing `identity:get_user(user_id=da2c6b0b94d4439e83cf89cc4878fefc)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:56.597437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3683698b-deab-4e41-9563-412267d1100a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:56.599328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 143/291] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:56 2022] GET /identity/v3/users/da2c6b0b94d4439e83cf89cc4878fefc => generated 297 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:56.604857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:56.650287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:56.713030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257e9610-4dbe-4292-b587-d8390200a9e7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:56.748043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a84b9-24ea-45e5-bf9b-4d5d6e445d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:56.748661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a84b9-24ea-45e5-bf9b-4d5d6e445d40 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:56.748945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a84b9-24ea-45e5-bf9b-4d5d6e445d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:56.749217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a84b9-24ea-45e5-bf9b-4d5d6e445d40 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:56.825760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a84b9-24ea-45e5-bf9b-4d5d6e445d40 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:56.827279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a84b9-24ea-45e5-bf9b-4d5d6e445d40 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:56.869892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:56.932006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 149/292] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:52:56.937660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3683698b-deab-4e41-9563-412267d1100a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:56.949898 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:57.041954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3683698b-deab-4e41-9563-412267d1100a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:57.076411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77821762-699f-4cfe-9e06-8032335f3951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:57.077088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77821762-699f-4cfe-9e06-8032335f3951 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:57.077367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77821762-699f-4cfe-9e06-8032335f3951 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:57.077656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77821762-699f-4cfe-9e06-8032335f3951 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:57.078828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77821762-699f-4cfe-9e06-8032335f3951 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:57.080251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77821762-699f-4cfe-9e06-8032335f3951 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:57.082185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 144/293] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:52:56 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:57.088401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a84b9-24ea-45e5-bf9b-4d5d6e445d40 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:57.203285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a84b9-24ea-45e5-bf9b-4d5d6e445d40 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:57.249187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843f093e-998e-42d2-8763-c26848027c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:57.249833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843f093e-998e-42d2-8763-c26848027c77 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:57.250117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843f093e-998e-42d2-8763-c26848027c77 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:57.250394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843f093e-998e-42d2-8763-c26848027c77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:57.251124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843f093e-998e-42d2-8763-c26848027c77 None admin] RBAC: Adding query filter params (role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16, user.id=da2c6b0b94d4439e83cf89cc4878fefc) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:57.251432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843f093e-998e-42d2-8763-c26848027c77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:57.252872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843f093e-998e-42d2-8763-c26848027c77 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:57.259041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 150/294] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:52:57 2022] GET /identity/v3/role_assignments?user.id=da2c6b0b94d4439e83cf89cc4878fefc&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:57.891070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207c2107-926e-4986-92a5-f45f5bd2fabb None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:57.891571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207c2107-926e-4986-92a5-f45f5bd2fabb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:57.891645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207c2107-926e-4986-92a5-f45f5bd2fabb None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:57.893191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 145/295] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:57.899274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e881f9-d66c-482d-9f44-a8fe3dbc7ae6 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:57.899816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e881f9-d66c-482d-9f44-a8fe3dbc7ae6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:57.900285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e881f9-d66c-482d-9f44-a8fe3dbc7ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:57.914139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:57.920182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-02e881f9-d66c-482d-9f44-a8fe3dbc7ae6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:57.921577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-02e881f9-d66c-482d-9f44-a8fe3dbc7ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:58.001758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41c971-7925-4eac-9a6a-247bed3289ee None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.002089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41c971-7925-4eac-9a6a-247bed3289ee None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.002320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41c971-7925-4eac-9a6a-247bed3289ee None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.003894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 146/297] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:58.004038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 151/297] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:58.008864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b45547-228b-4e28-b72a-f1978c1529db None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.009147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b45547-228b-4e28-b72a-f1978c1529db None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.009420 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b45547-228b-4e28-b72a-f1978c1529db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.027820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-49b45547-228b-4e28-b72a-f1978c1529db None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:58.028999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-49b45547-228b-4e28-b72a-f1978c1529db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:58.096495 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 147/298] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:58.282729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e881f9-d66c-482d-9f44-a8fe3dbc7ae6 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:58.372918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b45547-228b-4e28-b72a-f1978c1529db None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:58.387371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:58.388409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e881f9-d66c-482d-9f44-a8fe3dbc7ae6 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:58.423003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7285b3-105b-4962-b4a9-877e23263055 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:58.423642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7285b3-105b-4962-b4a9-877e23263055 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.423920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7285b3-105b-4962-b4a9-877e23263055 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.424190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7285b3-105b-4962-b4a9-877e23263055 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.427250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:58.436500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7285b3-105b-4962-b4a9-877e23263055 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-6e7285b3-105b-4962-b4a9-877e23263055 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:58.441964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:58.442836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:58.443645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 152/299] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:58 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:58.445222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7a560-eeef-42b8-a2ca-dc04e0b107a0 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.445702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7a560-eeef-42b8-a2ca-dc04e0b107a0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.445915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7a560-eeef-42b8-a2ca-dc04e0b107a0 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.447389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 153/300] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:58.450679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f7a560-eeef-42b8-a2ca-dc04e0b107a0 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:58.494817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b45547-228b-4e28-b72a-f1978c1529db None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:58.532490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aaa8af7-69ec-4b8e-82cb-c45d3842ed8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:58.533279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aaa8af7-69ec-4b8e-82cb-c45d3842ed8f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.533960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aaa8af7-69ec-4b8e-82cb-c45d3842ed8f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.534252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aaa8af7-69ec-4b8e-82cb-c45d3842ed8f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.535352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aaa8af7-69ec-4b8e-82cb-c45d3842ed8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:58.537438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aaa8af7-69ec-4b8e-82cb-c45d3842ed8f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:58.539351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 148/301] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:52:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:58.540290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f50fc92-0944-49b2-b7c5-8e9fca02a14f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.540570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f50fc92-0944-49b2-b7c5-8e9fca02a14f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.540839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f50fc92-0944-49b2-b7c5-8e9fca02a14f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.558980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1f50fc92-0944-49b2-b7c5-8e9fca02a14f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:58.560148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1f50fc92-0944-49b2-b7c5-8e9fca02a14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:58.564989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f7a560-eeef-42b8-a2ca-dc04e0b107a0 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:58.600765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:58.601375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.601682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.601952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.602603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:58.602899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:58.604258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:58.617554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 154/302] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:58 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:58.617554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:58.636475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 149/303] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:58.637334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f50fc92-0944-49b2-b7c5-8e9fca02a14f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:58.644887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:58.678306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:58.722699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b7d32b-285c-43ae-bad0-d20e8520b6f0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:58.749938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f50fc92-0944-49b2-b7c5-8e9fca02a14f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:58.758507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814997a0-bd23-4f6e-9f30-15241c539fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:58.759488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814997a0-bd23-4f6e-9f30-15241c539fba None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.759993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814997a0-bd23-4f6e-9f30-15241c539fba None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.760477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814997a0-bd23-4f6e-9f30-15241c539fba None admin] PATH_INFO: `/v3/groups` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.761453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814997a0-bd23-4f6e-9f30-15241c539fba None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:58.762980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814997a0-bd23-4f6e-9f30-15241c539fba None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:58.783833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cd1e0d-700e-4c12-a27a-6b775cb85e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:58.784458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd1e0d-700e-4c12-a27a-6b775cb85e55 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.784735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd1e0d-700e-4c12-a27a-6b775cb85e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.785003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cd1e0d-700e-4c12-a27a-6b775cb85e55 None admin] PATH_INFO: `/v3/users/4f0e741dcf294ea4aa5584167de22a93` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.799198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cd1e0d-700e-4c12-a27a-6b775cb85e55 None admin] RBAC: Authorizing `identity:get_user(user_id=4f0e741dcf294ea4aa5584167de22a93)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:58.800628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cd1e0d-700e-4c12-a27a-6b775cb85e55 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:58.802251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 150/304] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:58 2022] GET /identity/v3/users/4f0e741dcf294ea4aa5584167de22a93 => generated 293 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:58.807121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e146-9672-42b0-b6a5-1f33133f7d7a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.807257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e146-9672-42b0-b6a5-1f33133f7d7a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.807429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e146-9672-42b0-b6a5-1f33133f7d7a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.808694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 151/305] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:52:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:52:58.809429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f174e146-9672-42b0-b6a5-1f33133f7d7a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:58.828292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 155/306] 10.222.0.16 () {60 vars in 1134 bytes} [Thu Aug 25 04:52:58 2022] POST /identity/v3/groups => generated 229 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:52:58.835082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52365b76-8b35-4fbe-b23c-27af50fa2fcf None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.835082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52365b76-8b35-4fbe-b23c-27af50fa2fcf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.835082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52365b76-8b35-4fbe-b23c-27af50fa2fcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.858443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-52365b76-8b35-4fbe-b23c-27af50fa2fcf None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:52:58.860144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-52365b76-8b35-4fbe-b23c-27af50fa2fcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:52:58.891985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:58.939785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f174e146-9672-42b0-b6a5-1f33133f7d7a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:58.961634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 156/307] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:52:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:52:58.961634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52365b76-8b35-4fbe-b23c-27af50fa2fcf None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:58.977467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:58.978320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:58.978770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:58.979044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:58.980589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] RBAC: Adding query filter params (role.id=4ae9fa85613b4396ba3b5e4c3eff1cff, user.id=4f0e741dcf294ea4aa5584167de22a93, scope.system=all) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:58.980903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:58.982805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:58.989921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 152/308] 10.222.0.16 () {56 vars in 1308 bytes} [Thu Aug 25 04:52:58 2022] GET /identity/v3/role_assignments?user.id=4f0e741dcf294ea4aa5584167de22a93&scope.system=all&role.id=4ae9fa85613b4396ba3b5e4c3eff1cff => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:59.028090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:59.085142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52365b76-8b35-4fbe-b23c-27af50fa2fcf None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:59.125557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0071cd91-4a1c-4afc-a44c-561eb8f8f09e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:59.126245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0071cd91-4a1c-4afc-a44c-561eb8f8f09e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:59.126527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0071cd91-4a1c-4afc-a44c-561eb8f8f09e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:59.126804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0071cd91-4a1c-4afc-a44c-561eb8f8f09e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:59.139124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0071cd91-4a1c-4afc-a44c-561eb8f8f09e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-0071cd91-4a1c-4afc-a44c-561eb8f8f09e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:59.144630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:59.145536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:59.146386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 157/309] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:52:59.151919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:59.239727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0071cd91-4a1c-4afc-a44c-561eb8f8f09e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:59.260309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd63a10-eab7-4bfd-8da0-2714703a1f77 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:59.287470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:59.297354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:59.298272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:59.298717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:59.301574 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:59.301574 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:59.301574 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:59.301964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:59.312025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 153/310] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:59.318387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:59.352140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0071cd91-4a1c-4afc-a44c-561eb8f8f09e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:59.370205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:59.389830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6cf20-aa10-4639-ba80-7381bdf5aa30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:59.390578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6cf20-aa10-4639-ba80-7381bdf5aa30 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:59.390922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6cf20-aa10-4639-ba80-7381bdf5aa30 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:59.391688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6cf20-aa10-4639-ba80-7381bdf5aa30 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:59.403831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a6cf20-aa10-4639-ba80-7381bdf5aa30 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-37a6cf20-aa10-4639-ba80-7381bdf5aa30 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:52:59.409497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:52:59.410475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:52:59.411259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 158/311] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:52:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:52:59.416764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6cf20-aa10-4639-ba80-7381bdf5aa30 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:59.435340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bd4e5a-a0c5-42cf-843b-382648e3d00e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:59.483960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88093598-7a37-4dad-9860-e08d64518903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:59.484980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88093598-7a37-4dad-9860-e08d64518903 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:59.484980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88093598-7a37-4dad-9860-e08d64518903 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:59.485234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88093598-7a37-4dad-9860-e08d64518903 None admin] PATH_INFO: `/v3/users/18e834b69a2844ff9d98a8dda3c3b110` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:59.486535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88093598-7a37-4dad-9860-e08d64518903 None admin] RBAC: Authorizing `identity:get_user(user_id=18e834b69a2844ff9d98a8dda3c3b110)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:59.488029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88093598-7a37-4dad-9860-e08d64518903 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:59.494766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 154/312] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:59 2022] GET /identity/v3/users/18e834b69a2844ff9d98a8dda3c3b110 => generated 293 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:59.496348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88093598-7a37-4dad-9860-e08d64518903 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:59.543067 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6cf20-aa10-4639-ba80-7381bdf5aa30 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:59.554777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:59.579484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:59.580459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:59.580908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:59.581360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:59.582205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:52:59.582705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:59.584289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:59.593652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 159/313] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:52:59 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:52:59.599120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:59.607881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88093598-7a37-4dad-9860-e08d64518903 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:59.636503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:59.653146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdf9323-fa61-4c36-9c05-3e5448bb1f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:59.654455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdf9323-fa61-4c36-9c05-3e5448bb1f83 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:59.655203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdf9323-fa61-4c36-9c05-3e5448bb1f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:59.655722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdf9323-fa61-4c36-9c05-3e5448bb1f83 None admin] PATH_INFO: `/v3/system/users/18e834b69a2844ff9d98a8dda3c3b110/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:59.657892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdf9323-fa61-4c36-9c05-3e5448bb1f83 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=18e834b69a2844ff9d98a8dda3c3b110, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:59.659915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdf9323-fa61-4c36-9c05-3e5448bb1f83 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:59.716707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b592ccab-4986-4ef4-a08a-8c22cfec7c7b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:59.744559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 155/314] 10.222.0.16 () {58 vars in 1334 bytes} [Thu Aug 25 04:52:59 2022] PUT /identity/v3/system/users/18e834b69a2844ff9d98a8dda3c3b110/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:52:59.769141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7745d146-077f-425e-88f5-312c538752e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:59.769827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745d146-077f-425e-88f5-312c538752e8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:59.770154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745d146-077f-425e-88f5-312c538752e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:59.770478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7745d146-077f-425e-88f5-312c538752e8 None admin] PATH_INFO: `/v3/users/2e8c1ee74cba40f794834d08976d59d4` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:52:59.771920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7745d146-077f-425e-88f5-312c538752e8 None admin] RBAC: Authorizing `identity:get_user(user_id=2e8c1ee74cba40f794834d08976d59d4)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:52:59.773409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7745d146-077f-425e-88f5-312c538752e8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:52:59.775676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 160/315] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:52:59 2022] GET /identity/v3/users/2e8c1ee74cba40f794834d08976d59d4 => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:52:59.782148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdf9323-fa61-4c36-9c05-3e5448bb1f83 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:52:59.897486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdf9323-fa61-4c36-9c05-3e5448bb1f83 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:52:59.919358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:52:59.933847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc545109-256b-482f-adec-9b4d0b2654bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:52:59.934459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc545109-256b-482f-adec-9b4d0b2654bb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:52:59.934743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc545109-256b-482f-adec-9b4d0b2654bb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:52:59.935185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc545109-256b-482f-adec-9b4d0b2654bb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:00.009090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc545109-256b-482f-adec-9b4d0b2654bb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:00.010593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc545109-256b-482f-adec-9b4d0b2654bb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:00.121813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 156/316] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:52:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:00.123425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7745d146-077f-425e-88f5-312c538752e8 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:00.234204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:00.238848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7745d146-077f-425e-88f5-312c538752e8 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:00.277291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3884284-bccf-42b1-adc4-7ede82ccaa3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:00.277969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3884284-bccf-42b1-adc4-7ede82ccaa3a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:00.278247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3884284-bccf-42b1-adc4-7ede82ccaa3a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:00.278607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3884284-bccf-42b1-adc4-7ede82ccaa3a None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:00.279694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3884284-bccf-42b1-adc4-7ede82ccaa3a None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:00.281401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3884284-bccf-42b1-adc4-7ede82ccaa3a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:00.283855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 161/317] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:00 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:00.289428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc545109-256b-482f-adec-9b4d0b2654bb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:00.311529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:00.394055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc545109-256b-482f-adec-9b4d0b2654bb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:00.429549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9a87a7-fbf5-47bd-9307-85547d1bcc73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:00.430492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a87a7-fbf5-47bd-9307-85547d1bcc73 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:00.431033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a87a7-fbf5-47bd-9307-85547d1bcc73 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:00.431474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9a87a7-fbf5-47bd-9307-85547d1bcc73 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:00.432379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9a87a7-fbf5-47bd-9307-85547d1bcc73 None admin] RBAC: Adding query filter params (role.id=4ae9fa85613b4396ba3b5e4c3eff1cff, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16, user.id=2e8c1ee74cba40f794834d08976d59d4) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:00.432879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9a87a7-fbf5-47bd-9307-85547d1bcc73 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:00.434496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9a87a7-fbf5-47bd-9307-85547d1bcc73 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:00.441151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 157/318] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:00 2022] GET /identity/v3/role_assignments?user.id=2e8c1ee74cba40f794834d08976d59d4&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=4ae9fa85613b4396ba3b5e4c3eff1cff => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:00.849415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea307816-82d8-41c9-a801-c57c1f46c3fe None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:00.849983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea307816-82d8-41c9-a801-c57c1f46c3fe None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:00.849983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea307816-82d8-41c9-a801-c57c1f46c3fe None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:00.851371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 162/319] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:00.857105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03e1f8-4607-437e-8a99-4b01e9221d0b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:00.857398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03e1f8-4607-437e-8a99-4b01e9221d0b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:00.857710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03e1f8-4607-437e-8a99-4b01e9221d0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:00.875765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3b03e1f8-4607-437e-8a99-4b01e9221d0b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:00.876998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3b03e1f8-4607-437e-8a99-4b01e9221d0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:00.947267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 158/320] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:00.977648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad237e-4546-4bb8-854f-fbf95850ca0a None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:00.977931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad237e-4546-4bb8-854f-fbf95850ca0a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:00.978206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad237e-4546-4bb8-854f-fbf95850ca0a None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:00.979731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 163/321] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:00.984511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcaa0d-82d6-456c-b4fd-d72c6144a465 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:00.984799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcaa0d-82d6-456c-b4fd-d72c6144a465 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:00.985071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fcaa0d-82d6-456c-b4fd-d72c6144a465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.003153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-86fcaa0d-82d6-456c-b4fd-d72c6144a465 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:01.004308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-86fcaa0d-82d6-456c-b4fd-d72c6144a465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:01.013377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:01.068991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 159/322] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:01.220156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ad237e-4546-4bb8-854f-fbf95850ca0a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:01.322751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ad237e-4546-4bb8-854f-fbf95850ca0a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:01.335849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fcaa0d-82d6-456c-b4fd-d72c6144a465 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:01.358713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed85eae5-fe55-4d0d-9dce-ef99314d89be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:01.359352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed85eae5-fe55-4d0d-9dce-ef99314d89be None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:01.359633 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed85eae5-fe55-4d0d-9dce-ef99314d89be None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:01.359952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed85eae5-fe55-4d0d-9dce-ef99314d89be None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.361013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed85eae5-fe55-4d0d-9dce-ef99314d89be None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:01.362486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed85eae5-fe55-4d0d-9dce-ef99314d89be None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:01.364240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 164/323] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:01.369497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed85eae5-fe55-4d0d-9dce-ef99314d89be None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:01.404423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:01.439064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fcaa0d-82d6-456c-b4fd-d72c6144a465 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:01.472993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eef3b2f-684e-440b-b3a3-a62e33a81205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:01.473672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef3b2f-684e-440b-b3a3-a62e33a81205 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:01.473952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef3b2f-684e-440b-b3a3-a62e33a81205 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:01.474219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef3b2f-684e-440b-b3a3-a62e33a81205 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.475303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed85eae5-fe55-4d0d-9dce-ef99314d89be None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:01.485762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eef3b2f-684e-440b-b3a3-a62e33a81205 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7eef3b2f-684e-440b-b3a3-a62e33a81205 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:01.490539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:01.491427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:01.492591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 160/324] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:01.497782 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eef3b2f-684e-440b-b3a3-a62e33a81205 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:01.501799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:01.515676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e76a9a-2a67-4e77-ac20-1557f5184a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:01.516293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e76a9a-2a67-4e77-ac20-1557f5184a22 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:01.519300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e76a9a-2a67-4e77-ac20-1557f5184a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:01.519300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e76a9a-2a67-4e77-ac20-1557f5184a22 None admin] PATH_INFO: `/v3/groups` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.519300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e76a9a-2a67-4e77-ac20-1557f5184a22 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:01.519300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e76a9a-2a67-4e77-ac20-1557f5184a22 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:01.599131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eef3b2f-684e-440b-b3a3-a62e33a81205 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:01.623405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 165/325] 10.222.0.16 () {60 vars in 1134 bytes} [Thu Aug 25 04:53:01 2022] POST /identity/v3/groups => generated 226 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:53:01.637839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8bed10-4930-4fab-ab96-5a2efcb8e531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:01.638467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8bed10-4930-4fab-ab96-5a2efcb8e531 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:01.638745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8bed10-4930-4fab-ab96-5a2efcb8e531 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:01.639227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8bed10-4930-4fab-ab96-5a2efcb8e531 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.639850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8bed10-4930-4fab-ab96-5a2efcb8e531 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:01.640148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8bed10-4930-4fab-ab96-5a2efcb8e531 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:01.641557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8bed10-4930-4fab-ab96-5a2efcb8e531 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:01.654007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 161/326] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:01 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:01.658051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e76a9a-2a67-4e77-ac20-1557f5184a22 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:01.714172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f191be7e-46d3-4243-b786-a7efceba4579 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:01.714463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f191be7e-46d3-4243-b786-a7efceba4579 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:01.714738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f191be7e-46d3-4243-b786-a7efceba4579 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.716238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 162/327] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:53:01.721628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc7be7c-bd63-4e75-9f7d-add2dc33ae17 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:01.722022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc7be7c-bd63-4e75-9f7d-add2dc33ae17 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:01.722220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc7be7c-bd63-4e75-9f7d-add2dc33ae17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.741723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3bc7be7c-bd63-4e75-9f7d-add2dc33ae17 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:01.742832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3bc7be7c-bd63-4e75-9f7d-add2dc33ae17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:01.767561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:01.775675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e76a9a-2a67-4e77-ac20-1557f5184a22 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:01.813851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a951bae0-a8f9-454f-b9ce-9dcefb31c457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:01.814403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:01.818995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a951bae0-a8f9-454f-b9ce-9dcefb31c457 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:01.818995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a951bae0-a8f9-454f-b9ce-9dcefb31c457 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:01.818995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a951bae0-a8f9-454f-b9ce-9dcefb31c457 None admin] PATH_INFO: `/v3/users/4f0e741dcf294ea4aa5584167de22a93` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.818995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a951bae0-a8f9-454f-b9ce-9dcefb31c457 None admin] RBAC: Authorizing `identity:get_user(user_id=4f0e741dcf294ea4aa5584167de22a93)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:01.818995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a951bae0-a8f9-454f-b9ce-9dcefb31c457 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:01.819530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 166/328] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:53:01 2022] GET /identity/v3/users/4f0e741dcf294ea4aa5584167de22a93 => generated 293 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:01.822046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 163/329] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:01.825148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a951bae0-a8f9-454f-b9ce-9dcefb31c457 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:01.933890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a951bae0-a8f9-454f-b9ce-9dcefb31c457 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:01.970647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd08607-2584-4658-a843-1034a748ebcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:01.971657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd08607-2584-4658-a843-1034a748ebcd None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:01.972166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd08607-2584-4658-a843-1034a748ebcd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:01.972448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd08607-2584-4658-a843-1034a748ebcd None admin] PATH_INFO: `/v3/system/users/4f0e741dcf294ea4aa5584167de22a93/roles/4ae9fa85613b4396ba3b5e4c3eff1cff` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:01.973898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd08607-2584-4658-a843-1034a748ebcd None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=4f0e741dcf294ea4aa5584167de22a93, role_id=4ae9fa85613b4396ba3b5e4c3eff1cff)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:01.975271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd08607-2584-4658-a843-1034a748ebcd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:02.058742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 167/330] 10.222.0.16 () {58 vars in 1334 bytes} [Thu Aug 25 04:53:01 2022] PUT /identity/v3/system/users/4f0e741dcf294ea4aa5584167de22a93/roles/4ae9fa85613b4396ba3b5e4c3eff1cff => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:02.094521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc7be7c-bd63-4e75-9f7d-add2dc33ae17 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:02.217313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc7be7c-bd63-4e75-9f7d-add2dc33ae17 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:02.263680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f2eda5-995d-4c6f-bbe7-1205f8ac98c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:02.264404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f2eda5-995d-4c6f-bbe7-1205f8ac98c5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:02.264689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f2eda5-995d-4c6f-bbe7-1205f8ac98c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:02.264965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f2eda5-995d-4c6f-bbe7-1205f8ac98c5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:02.276136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f2eda5-995d-4c6f-bbe7-1205f8ac98c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d0f2eda5-995d-4c6f-bbe7-1205f8ac98c5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:02.281236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:02.282131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:02.282920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 164/331] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:02.288128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd08607-2584-4658-a843-1034a748ebcd None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:02.318803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:02.433152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd08607-2584-4658-a843-1034a748ebcd None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:02.475238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:02.475870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:02.476148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:02.476422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:02.477093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:02.477399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:02.478818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:02.490973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 168/332] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:02.497187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f2eda5-995d-4c6f-bbe7-1205f8ac98c5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:02.580719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:02.637692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f2eda5-995d-4c6f-bbe7-1205f8ac98c5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:02.678565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab789c2d-50ba-4917-88da-54ff8da8aab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:02.679205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab789c2d-50ba-4917-88da-54ff8da8aab4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:02.679481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab789c2d-50ba-4917-88da-54ff8da8aab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:02.679751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab789c2d-50ba-4917-88da-54ff8da8aab4 None admin] PATH_INFO: `/v3/users/18e834b69a2844ff9d98a8dda3c3b110` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:02.683312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab789c2d-50ba-4917-88da-54ff8da8aab4 None admin] RBAC: Authorizing `identity:get_user(user_id=18e834b69a2844ff9d98a8dda3c3b110)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:02.684747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab789c2d-50ba-4917-88da-54ff8da8aab4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:02.686678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 165/333] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:53:02 2022] GET /identity/v3/users/18e834b69a2844ff9d98a8dda3c3b110 => generated 293 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:02.692501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:02.821256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:02.841080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73c2e45-cf0a-4ff4-a80c-bd29677e32d1 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:02.888038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9b616c-5d7e-45d5-aec0-4a1ccb845d3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:02.888894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b616c-5d7e-45d5-aec0-4a1ccb845d3b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:02.889315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b616c-5d7e-45d5-aec0-4a1ccb845d3b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:02.889779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9b616c-5d7e-45d5-aec0-4a1ccb845d3b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:02.890645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9b616c-5d7e-45d5-aec0-4a1ccb845d3b None admin] RBAC: Adding query filter params (role.id=0940d9778fb041fbaf7c4f4317feb2d5, user.id=18e834b69a2844ff9d98a8dda3c3b110, scope.system=all) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:02.891096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9b616c-5d7e-45d5-aec0-4a1ccb845d3b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:02.892688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9b616c-5d7e-45d5-aec0-4a1ccb845d3b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:02.908955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 169/334] 10.222.0.16 () {56 vars in 1308 bytes} [Thu Aug 25 04:53:02 2022] GET /identity/v3/role_assignments?user.id=18e834b69a2844ff9d98a8dda3c3b110&scope.system=all&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 510 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:03.952355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7f392b-35e1-4231-b228-b846a5c83bc7 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:03.952863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7f392b-35e1-4231-b228-b846a5c83bc7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:03.952940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7f392b-35e1-4231-b228-b846a5c83bc7 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:03.954514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 166/335] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:03.960282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b430c328-193f-4392-8a37-ab9aea24835a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:03.961119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b430c328-193f-4392-8a37-ab9aea24835a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:03.961611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b430c328-193f-4392-8a37-ab9aea24835a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:03.986686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-b430c328-193f-4392-8a37-ab9aea24835a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:03.988349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b430c328-193f-4392-8a37-ab9aea24835a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:04.075908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 170/336] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:04.420103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f50078-042b-4952-b61b-73e183baa4af None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:04.420795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f50078-042b-4952-b61b-73e183baa4af None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:04.421270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f50078-042b-4952-b61b-73e183baa4af None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:04.423193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 167/337] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:04.428387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e5f569-ffd6-4a1f-9ba2-dc8212802859 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:04.428911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e5f569-ffd6-4a1f-9ba2-dc8212802859 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:04.429375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e5f569-ffd6-4a1f-9ba2-dc8212802859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:04.454966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-84e5f569-ffd6-4a1f-9ba2-dc8212802859 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:04.456531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-84e5f569-ffd6-4a1f-9ba2-dc8212802859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:04.498336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a674f306-e902-4485-b78d-3f53280052ff None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:04.498919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a674f306-e902-4485-b78d-3f53280052ff None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:04.499383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a674f306-e902-4485-b78d-3f53280052ff None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:04.500902 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 168/338] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:53:04.507498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201011b2-c4f1-49e5-b1b1-db2eafdc5049 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:04.508089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201011b2-c4f1-49e5-b1b1-db2eafdc5049 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:04.508557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201011b2-c4f1-49e5-b1b1-db2eafdc5049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:04.547249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 171/339] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:04.548921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-201011b2-c4f1-49e5-b1b1-db2eafdc5049 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:04.550159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-201011b2-c4f1-49e5-b1b1-db2eafdc5049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:04.571258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e5f569-ffd6-4a1f-9ba2-dc8212802859 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:04.601978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:04.668305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 169/340] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:04.682936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e5f569-ffd6-4a1f-9ba2-dc8212802859 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:04.719294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf17977-8c42-4665-992b-172f47426e86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:04.719931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf17977-8c42-4665-992b-172f47426e86 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:04.720206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf17977-8c42-4665-992b-172f47426e86 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:04.720472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf17977-8c42-4665-992b-172f47426e86 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:04.730932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf17977-8c42-4665-992b-172f47426e86 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-edf17977-8c42-4665-992b-172f47426e86 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:04.736950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:04.737853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:04.738643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 172/341] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:04.745926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201011b2-c4f1-49e5-b1b1-db2eafdc5049 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:04.832419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:04.882016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201011b2-c4f1-49e5-b1b1-db2eafdc5049 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:04.922998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:04.923929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:04.924380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:04.924833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:04.925688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:04.926161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:04.927695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:04.936922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf17977-8c42-4665-992b-172f47426e86 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:04.938986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 170/342] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:04.945365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:05.014937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:05.050070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf17977-8c42-4665-992b-172f47426e86 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:05.086186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23510b3e-e3f2-44d3-ba77-096ded8fe100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:05.087298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23510b3e-e3f2-44d3-ba77-096ded8fe100 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.087721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23510b3e-e3f2-44d3-ba77-096ded8fe100 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.088240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23510b3e-e3f2-44d3-ba77-096ded8fe100 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.099560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23510b3e-e3f2-44d3-ba77-096ded8fe100 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-23510b3e-e3f2-44d3-ba77-096ded8fe100 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:05.105730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:05.106724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 25 04:53:05.107518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:05.113848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 173/343] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:05.115451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23510b3e-e3f2-44d3-ba77-096ded8fe100 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:05.122984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81196d83-5eb1-4b06-9516-8ccaa2cde51f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:05.144100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:05.167289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbd949-bbc6-4d3a-a818-0326908b14fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:05.167735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbd949-bbc6-4d3a-a818-0326908b14fe None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.167954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbd949-bbc6-4d3a-a818-0326908b14fe None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.168227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbd949-bbc6-4d3a-a818-0326908b14fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.226094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23510b3e-e3f2-44d3-ba77-096ded8fe100 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:05.242893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cbd949-bbc6-4d3a-a818-0326908b14fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:05.244348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cbd949-bbc6-4d3a-a818-0326908b14fe None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:05.258963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:05.268558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:05.269556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.270001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.270409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.271210 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] RBAC: Adding query filter params (name=reader) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:05.271746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:05.273280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:05.283476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 174/344] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:05.284569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:05.357354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 171/345] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 25 04:53:05.359873 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbd949-bbc6-4d3a-a818-0326908b14fe None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:05.378986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:05.430310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1493683f-2d0e-4a99-be5c-f657826b1dcf None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:05.474013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a043450-ef00-4aeb-b0f8-60b68a436804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:05.474892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a043450-ef00-4aeb-b0f8-60b68a436804 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.475308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a043450-ef00-4aeb-b0f8-60b68a436804 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.475751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a043450-ef00-4aeb-b0f8-60b68a436804 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.488466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a043450-ef00-4aeb-b0f8-60b68a436804 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-4a043450-ef00-4aeb-b0f8-60b68a436804 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:05.495479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:05.496644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:05.497671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:05.500135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 175/346] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:53:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:05.501074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a043450-ef00-4aeb-b0f8-60b68a436804 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:05.542172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbd949-bbc6-4d3a-a818-0326908b14fe None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:05.566318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:05.624404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a043450-ef00-4aeb-b0f8-60b68a436804 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:05.642682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f22827-98c1-4817-a536-014e54699538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:05.643436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f22827-98c1-4817-a536-014e54699538 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.643788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f22827-98c1-4817-a536-014e54699538 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.644129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f22827-98c1-4817-a536-014e54699538 None admin] PATH_INFO: `/v3/users/4f0e741dcf294ea4aa5584167de22a93` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.645378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f22827-98c1-4817-a536-014e54699538 None admin] RBAC: Authorizing `identity:get_user(user_id=4f0e741dcf294ea4aa5584167de22a93)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:05.646922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f22827-98c1-4817-a536-014e54699538 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:05.648877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 172/347] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:53:05 2022] GET /identity/v3/users/4f0e741dcf294ea4aa5584167de22a93 => generated 293 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:05.649752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f22827-98c1-4817-a536-014e54699538 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:05.671694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191a0ee4-c594-400b-8e44-d402c43f3544 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:05.672429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a0ee4-c594-400b-8e44-d402c43f3544 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.672776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a0ee4-c594-400b-8e44-d402c43f3544 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.673119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a0ee4-c594-400b-8e44-d402c43f3544 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.674355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191a0ee4-c594-400b-8e44-d402c43f3544 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:05.675919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191a0ee4-c594-400b-8e44-d402c43f3544 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:05.677645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 176/348] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:05 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:05.679037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7e65c5-2cd6-405c-9d64-015e7639840d None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.679146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7e65c5-2cd6-405c-9d64-015e7639840d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.679445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7e65c5-2cd6-405c-9d64-015e7639840d None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.680715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 177/349] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:05.681477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7e65c5-2cd6-405c-9d64-015e7639840d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:05.766630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:05.784098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f22827-98c1-4817-a536-014e54699538 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:05.811178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7e65c5-2cd6-405c-9d64-015e7639840d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:05.823970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:05.824709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.825054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.825403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.826322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:05.826695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:05.828137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:05.836868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:05.843168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 173/350] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:05.844707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:05.862382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef706fc8-327e-4dc4-bb3c-56d2801ba15f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:05.863613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef706fc8-327e-4dc4-bb3c-56d2801ba15f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.863863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef706fc8-327e-4dc4-bb3c-56d2801ba15f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.864372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef706fc8-327e-4dc4-bb3c-56d2801ba15f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.865231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef706fc8-327e-4dc4-bb3c-56d2801ba15f None admin] RBAC: Adding query filter params (role.id=4ae9fa85613b4396ba3b5e4c3eff1cff, user.id=4f0e741dcf294ea4aa5584167de22a93, scope.system=all) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:05.865469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef706fc8-327e-4dc4-bb3c-56d2801ba15f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:05.866958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef706fc8-327e-4dc4-bb3c-56d2801ba15f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:05.880233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 178/351] 10.222.0.16 () {56 vars in 1308 bytes} [Thu Aug 25 04:53:05 2022] GET /identity/v3/role_assignments?user.id=4f0e741dcf294ea4aa5584167de22a93&scope.system=all&role.id=4ae9fa85613b4396ba3b5e4c3eff1cff => generated 510 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:05.881459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540e33bb-ec42-4c91-bb70-14d4cb36cb63 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:05.886158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540e33bb-ec42-4c91-bb70-14d4cb36cb63 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:05.886475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540e33bb-ec42-4c91-bb70-14d4cb36cb63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:05.911455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-540e33bb-ec42-4c91-bb70-14d4cb36cb63 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:05.912482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-540e33bb-ec42-4c91-bb70-14d4cb36cb63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:05.962253 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4089cbe5-f6b5-468a-93e9-91d259964a18 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:06.006812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75112d6-fc83-4d3e-82e1-a656e0f0837f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:06.008657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75112d6-fc83-4d3e-82e1-a656e0f0837f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.010157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75112d6-fc83-4d3e-82e1-a656e0f0837f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.011199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75112d6-fc83-4d3e-82e1-a656e0f0837f None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.024960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75112d6-fc83-4d3e-82e1-a656e0f0837f None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:06.028155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75112d6-fc83-4d3e-82e1-a656e0f0837f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:06.031557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 174/352] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:05 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:06.032818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75112d6-fc83-4d3e-82e1-a656e0f0837f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:06.051100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 179/353] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:06.053127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d704b-5f39-4709-b532-4afb5a4d5cd9 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.053716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d704b-5f39-4709-b532-4afb5a4d5cd9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.054187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d704b-5f39-4709-b532-4afb5a4d5cd9 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.055957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:06.058455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 180/354] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:06.059759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329d704b-5f39-4709-b532-4afb5a4d5cd9 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:06.202198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75112d6-fc83-4d3e-82e1-a656e0f0837f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:06.243140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329d704b-5f39-4709-b532-4afb5a4d5cd9 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:06.246653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:06.258198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff2972a-608c-4a88-ad77-943bb7575bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:06.265739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff2972a-608c-4a88-ad77-943bb7575bd7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.265739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff2972a-608c-4a88-ad77-943bb7575bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.265739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff2972a-608c-4a88-ad77-943bb7575bd7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.299109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42fbf2-c445-42b3-beba-c413bb03b4cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:06.299716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42fbf2-c445-42b3-beba-c413bb03b4cf None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.299994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42fbf2-c445-42b3-beba-c413bb03b4cf None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.300262 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42fbf2-c445-42b3-beba-c413bb03b4cf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.300980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42fbf2-c445-42b3-beba-c413bb03b4cf None admin] RBAC: Adding query filter params (group.id=80da8e2dc5874812b1d79ec0374e05fd, role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=8419cf9f31bb4753946ad654d60e9e26) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:06.301283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42fbf2-c445-42b3-beba-c413bb03b4cf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:06.303110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42fbf2-c445-42b3-beba-c413bb03b4cf None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:06.320082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 181/355] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:06 2022] GET /identity/v3/role_assignments?group.id=80da8e2dc5874812b1d79ec0374e05fd&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 260 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:06.325721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d957e829-6c5a-44de-a67d-93ae18e266db None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.326699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d957e829-6c5a-44de-a67d-93ae18e266db None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.326699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d957e829-6c5a-44de-a67d-93ae18e266db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.351315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-d957e829-6c5a-44de-a67d-93ae18e266db None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:06.352840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d957e829-6c5a-44de-a67d-93ae18e266db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:06.361646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff2972a-608c-4a88-ad77-943bb7575bd7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:06.361646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff2972a-608c-4a88-ad77-943bb7575bd7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:06.376369 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:06.505776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 182/356] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:06.507106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9535816-9faa-44ba-90ac-41fab2a4d655 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.507396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9535816-9faa-44ba-90ac-41fab2a4d655 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.507673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9535816-9faa-44ba-90ac-41fab2a4d655 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.510063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 183/357] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:06.515795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd6a9f5-6f49-4e34-a0f6-20856b8122e8 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.516091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd6a9f5-6f49-4e34-a0f6-20856b8122e8 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.516363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd6a9f5-6f49-4e34-a0f6-20856b8122e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.523795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 175/358] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 491 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:06.529486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff2972a-608c-4a88-ad77-943bb7575bd7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:06.542913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-9bd6a9f5-6f49-4e34-a0f6-20856b8122e8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:06.544541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9bd6a9f5-6f49-4e34-a0f6-20856b8122e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:06.645621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 184/359] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:06.645621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd6a9f5-6f49-4e34-a0f6-20856b8122e8 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:06.662238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff2972a-608c-4a88-ad77-943bb7575bd7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:06.681364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:06.698127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:06.746329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09db052-6373-4a1e-bd1c-730d97ea7366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:06.747034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09db052-6373-4a1e-bd1c-730d97ea7366 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.747319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09db052-6373-4a1e-bd1c-730d97ea7366 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.747595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09db052-6373-4a1e-bd1c-730d97ea7366 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.755374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd6a9f5-6f49-4e34-a0f6-20856b8122e8 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:06.770639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09db052-6373-4a1e-bd1c-730d97ea7366 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b09db052-6373-4a1e-bd1c-730d97ea7366 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:06.775505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:06.776426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:06.777323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 176/360] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:06.784334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09db052-6373-4a1e-bd1c-730d97ea7366 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:06.799767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff06afd-94e8-4d32-ac03-4979ffab8c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:06.801092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff06afd-94e8-4d32-ac03-4979ffab8c44 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.801514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff06afd-94e8-4d32-ac03-4979ffab8c44 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.801976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff06afd-94e8-4d32-ac03-4979ffab8c44 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.804481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff06afd-94e8-4d32-ac03-4979ffab8c44 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:06.806554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff06afd-94e8-4d32-ac03-4979ffab8c44 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:06.809250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 185/361] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:06 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:06.815306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff06afd-94e8-4d32-ac03-4979ffab8c44 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:06.909695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09db052-6373-4a1e-bd1c-730d97ea7366 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:06.920153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:06.926504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff06afd-94e8-4d32-ac03-4979ffab8c44 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:06.957572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:06.958897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.959367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.959835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.960720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:06.961209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:06.963186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5238d412-486e-4dc6-adc5-0135be3c8ae1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:06.963996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:06.964377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238d412-486e-4dc6-adc5-0135be3c8ae1 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:06.964660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238d412-486e-4dc6-adc5-0135be3c8ae1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:06.964932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238d412-486e-4dc6-adc5-0135be3c8ae1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:06.974779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 177/362] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:06.976126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:06.976231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5238d412-486e-4dc6-adc5-0135be3c8ae1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-5238d412-486e-4dc6-adc5-0135be3c8ae1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:06.981579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:06.982483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:06.983279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 186/363] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:53:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:06.983935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5238d412-486e-4dc6-adc5-0135be3c8ae1 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.046110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:07.115264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5238d412-486e-4dc6-adc5-0135be3c8ae1 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:07.130356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40827820-4eec-44b6-9b43-42283b7293c6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:07.160728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badba9c3-5d6b-4f9b-9c8a-f382b2c37b64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:07.161344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badba9c3-5d6b-4f9b-9c8a-f382b2c37b64 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:07.161658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badba9c3-5d6b-4f9b-9c8a-f382b2c37b64 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:07.161927 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badba9c3-5d6b-4f9b-9c8a-f382b2c37b64 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:07.175518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac62964-73bf-4f82-9514-30b13922b684 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:07.176169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac62964-73bf-4f82-9514-30b13922b684 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:07.176447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac62964-73bf-4f82-9514-30b13922b684 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:07.176721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac62964-73bf-4f82-9514-30b13922b684 None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:07.177886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac62964-73bf-4f82-9514-30b13922b684 None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:07.179332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac62964-73bf-4f82-9514-30b13922b684 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:07.180998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 178/364] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:06 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:07.183039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac62964-73bf-4f82-9514-30b13922b684 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.194868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:07.252763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badba9c3-5d6b-4f9b-9c8a-f382b2c37b64 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:07.254585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badba9c3-5d6b-4f9b-9c8a-f382b2c37b64 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:07.307767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:07.312073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac62964-73bf-4f82-9514-30b13922b684 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:07.352212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:07.352830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:07.353106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:07.353374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:07.355425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:07.355425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:07.356365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:07.365950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 179/365] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:07.366775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.380923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 187/366] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:07.382636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badba9c3-5d6b-4f9b-9c8a-f382b2c37b64 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.490234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:07.501088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badba9c3-5d6b-4f9b-9c8a-f382b2c37b64 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:07.503890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c388c1b-fe0f-45e8-8581-8827fb3e4bed None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:07.544959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:07.545576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:07.545880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:07.546175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:07.547535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] RBAC: Adding query filter params (group.id=80da8e2dc5874812b1d79ec0374e05fd, role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=8419cf9f31bb4753946ad654d60e9e26) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:07.547535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:07.548699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:07.555286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 188/367] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:07 2022] GET /identity/v3/role_assignments?group.id=80da8e2dc5874812b1d79ec0374e05fd&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 260 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:07.555889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.562429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb44faa2-aff8-47cf-a6ae-832fa52dfa8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:07.563155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44faa2-aff8-47cf-a6ae-832fa52dfa8a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:07.563537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44faa2-aff8-47cf-a6ae-832fa52dfa8a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:07.564129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44faa2-aff8-47cf-a6ae-832fa52dfa8a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:07.582392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb44faa2-aff8-47cf-a6ae-832fa52dfa8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-cb44faa2-aff8-47cf-a6ae-832fa52dfa8a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:07.588200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:07.589308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:07.590180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 180/368] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:07.590895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb44faa2-aff8-47cf-a6ae-832fa52dfa8a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.621006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:07.708883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726dfa86-b97d-4af3-be69-db255d7f7f8b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:07.716240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb44faa2-aff8-47cf-a6ae-832fa52dfa8a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:07.734823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:07.751072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345f92d4-641b-4a80-a305-c2e481e2ead4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:07.751687 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f92d4-641b-4a80-a305-c2e481e2ead4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:07.751963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f92d4-641b-4a80-a305-c2e481e2ead4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:07.752227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f92d4-641b-4a80-a305-c2e481e2ead4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:07.771189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3a4fce-df5f-4968-80fe-907e710d9ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:07.772299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3a4fce-df5f-4968-80fe-907e710d9ed6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:07.772892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3a4fce-df5f-4968-80fe-907e710d9ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:07.773312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3a4fce-df5f-4968-80fe-907e710d9ed6 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:07.775186 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3a4fce-df5f-4968-80fe-907e710d9ed6 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:07.777864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3a4fce-df5f-4968-80fe-907e710d9ed6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:07.779759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 181/369] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:07 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:07.781203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3a4fce-df5f-4968-80fe-907e710d9ed6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.835728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345f92d4-641b-4a80-a305-c2e481e2ead4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:07.837427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345f92d4-641b-4a80-a305-c2e481e2ead4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:07.894124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:07.906332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3a4fce-df5f-4968-80fe-907e710d9ed6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:07.952148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:07.952766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:07.953037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:07.953306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:07.954004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:07.954301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:07.955712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:07.970092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 189/370] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:07.971243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345f92d4-641b-4a80-a305-c2e481e2ead4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.972136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 182/371] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:07.977624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:07.999698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:08.092887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345f92d4-641b-4a80-a305-c2e481e2ead4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:08.113150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59b46b4-a4da-43c4-9fd2-11cf0be6e87d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:08.130183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44229aeb-a257-4554-ae9c-fd769f5c998f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:08.131492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44229aeb-a257-4554-ae9c-fd769f5c998f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.131973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44229aeb-a257-4554-ae9c-fd769f5c998f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.132440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44229aeb-a257-4554-ae9c-fd769f5c998f None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.133793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44229aeb-a257-4554-ae9c-fd769f5c998f None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:08.135470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44229aeb-a257-4554-ae9c-fd769f5c998f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:08.137699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 190/372] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:07 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:08.138760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44229aeb-a257-4554-ae9c-fd769f5c998f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:08.161262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7aadb9-d766-4ee2-89ef-7230fd087caa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:08.161975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7aadb9-d766-4ee2-89ef-7230fd087caa None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.162301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7aadb9-d766-4ee2-89ef-7230fd087caa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.162575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7aadb9-d766-4ee2-89ef-7230fd087caa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.187449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:08.240310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:08.247849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7aadb9-d766-4ee2-89ef-7230fd087caa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:08.249361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7aadb9-d766-4ee2-89ef-7230fd087caa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:08.263103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44229aeb-a257-4554-ae9c-fd769f5c998f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:08.307371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a7fc61-b103-4689-b107-23196384432d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:08.308265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a7fc61-b103-4689-b107-23196384432d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.308690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a7fc61-b103-4689-b107-23196384432d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.309113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a7fc61-b103-4689-b107-23196384432d None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.310410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a7fc61-b103-4689-b107-23196384432d None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:08.312023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a7fc61-b103-4689-b107-23196384432d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:08.313815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 191/373] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:08 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:08.314696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a7fc61-b103-4689-b107-23196384432d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:08.383647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 183/374] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:08.385056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7aadb9-d766-4ee2-89ef-7230fd087caa None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:08.454090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:08.467852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a7fc61-b103-4689-b107-23196384432d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:08.512710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7aadb9-d766-4ee2-89ef-7230fd087caa None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:08.522108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:08.522744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.523362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.523644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.524375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] RBAC: Adding query filter params (group.id=80da8e2dc5874812b1d79ec0374e05fd, role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:08.525000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:08.527018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:08.534966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 192/375] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:08 2022] GET /identity/v3/role_assignments?group.id=80da8e2dc5874812b1d79ec0374e05fd&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 260 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:08.536181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:08.555986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e020af73-9fcd-4520-af77-0bd02dcf2f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:08.556641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020af73-9fcd-4520-af77-0bd02dcf2f9b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.556919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020af73-9fcd-4520-af77-0bd02dcf2f9b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.557744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e020af73-9fcd-4520-af77-0bd02dcf2f9b None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.558986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e020af73-9fcd-4520-af77-0bd02dcf2f9b None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:08.560502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e020af73-9fcd-4520-af77-0bd02dcf2f9b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:08.562502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 184/376] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:08 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:08.563709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aebbcf-e888-4da3-8b0e-12449fc0cf80 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.563988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aebbcf-e888-4da3-8b0e-12449fc0cf80 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.564258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3aebbcf-e888-4da3-8b0e-12449fc0cf80 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.569682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 185/377] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:08.576742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65662f99-627b-4e7f-a7f5-141188057e5c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.577904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65662f99-627b-4e7f-a7f5-141188057e5c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.578743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65662f99-627b-4e7f-a7f5-141188057e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.583680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:08.606279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-65662f99-627b-4e7f-a7f5-141188057e5c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:08.608475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-65662f99-627b-4e7f-a7f5-141188057e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:08.688070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b87d1-c3db-48df-b95a-f269f000ac34 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:08.708111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 186/378] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:08.709314 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65662f99-627b-4e7f-a7f5-141188057e5c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:08.758333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450972a2-4b05-44d6-a1ec-f39de8a41991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:08.758994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450972a2-4b05-44d6-a1ec-f39de8a41991 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.760965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450972a2-4b05-44d6-a1ec-f39de8a41991 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.761238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450972a2-4b05-44d6-a1ec-f39de8a41991 None admin] PATH_INFO: `/v3/projects/f65e42175ca249f08b4d6dc3b72146f4` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.764338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450972a2-4b05-44d6-a1ec-f39de8a41991 None admin] RBAC: Authorizing `identity:get_project(project_id=f65e42175ca249f08b4d6dc3b72146f4)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:08.767434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450972a2-4b05-44d6-a1ec-f39de8a41991 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:08.770000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 193/379] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:08 2022] GET /identity/v3/projects/f65e42175ca249f08b4d6dc3b72146f4 => generated 344 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:08.775522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450972a2-4b05-44d6-a1ec-f39de8a41991 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:08.785842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:08.860451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65662f99-627b-4e7f-a7f5-141188057e5c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:08.896521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:08.906232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:08.907149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.907658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.908140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.909087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] RBAC: Adding query filter params (group.id=80da8e2dc5874812b1d79ec0374e05fd, role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:08.909633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:08.911321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:08.911956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450972a2-4b05-44d6-a1ec-f39de8a41991 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:08.918381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 187/380] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:08 2022] GET /identity/v3/role_assignments?group.id=80da8e2dc5874812b1d79ec0374e05fd&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 260 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:08.951369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8640571-e91b-44df-bb22-c34c537e8f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:08.951943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8640571-e91b-44df-bb22-c34c537e8f11 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:08.952219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8640571-e91b-44df-bb22-c34c537e8f11 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:08.952490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8640571-e91b-44df-bb22-c34c537e8f11 None admin] PATH_INFO: `/v3/groups/452fb2080a4c42eebf62abc27555bb2e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:08.962287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8640571-e91b-44df-bb22-c34c537e8f11 None admin] RBAC: Authorizing `identity:get_group(group_id=452fb2080a4c42eebf62abc27555bb2e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:08.963816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8640571-e91b-44df-bb22-c34c537e8f11 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:08.965925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 194/381] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:08 2022] GET /identity/v3/groups/452fb2080a4c42eebf62abc27555bb2e => generated 229 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:08.971607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:09.025230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8640571-e91b-44df-bb22-c34c537e8f11 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:09.111288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0504bb36-879f-49b7-9342-7af279701bbe None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:09.129443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:09.152267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:09.153237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:09.153784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:09.154273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:09.155205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] RBAC: Adding query filter params (group.id=452fb2080a4c42eebf62abc27555bb2e, role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=f65e42175ca249f08b4d6dc3b72146f4) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:09.155747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:09.157570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:09.164532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 188/382] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:08 2022] GET /identity/v3/role_assignments?group.id=452fb2080a4c42eebf62abc27555bb2e&scope.project.id=f65e42175ca249f08b4d6dc3b72146f4&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 260 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:09.255753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8640571-e91b-44df-bb22-c34c537e8f11 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:09.305322 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d16dccb-a82a-41ac-bf35-40abf13526bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:09.305999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d16dccb-a82a-41ac-bf35-40abf13526bc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:09.306277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d16dccb-a82a-41ac-bf35-40abf13526bc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:09.306546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d16dccb-a82a-41ac-bf35-40abf13526bc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:09.323720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d16dccb-a82a-41ac-bf35-40abf13526bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-4d16dccb-a82a-41ac-bf35-40abf13526bc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:09.329222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:09.330149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:09.330988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 195/383] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:09.336313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:09.489614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782fe8a4-cb75-4a0b-be02-7256009f3df3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:09.529440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:09.530176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:09.530427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:09.530702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:09.531380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:09.531674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:09.533037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:09.542668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 189/384] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:09.548274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d16dccb-a82a-41ac-bf35-40abf13526bc None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:09.565075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:09.664145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d16dccb-a82a-41ac-bf35-40abf13526bc None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:09.706268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb042a-096f-4347-b501-85e06bf5a503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:09.706915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb042a-096f-4347-b501-85e06bf5a503 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:09.707194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb042a-096f-4347-b501-85e06bf5a503 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:09.707466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb042a-096f-4347-b501-85e06bf5a503 None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:09.708615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fb042a-096f-4347-b501-85e06bf5a503 None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:09.710101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fb042a-096f-4347-b501-85e06bf5a503 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:09.712927 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 196/385] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:09 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:09.717498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:09.766444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:09.831661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e19a2f-8fcd-4931-97e9-a6771dcd6904 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:09.870742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff52251c-d5de-4d18-8f65-7e724a2474bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:09.871380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52251c-d5de-4d18-8f65-7e724a2474bc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:09.871659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52251c-d5de-4d18-8f65-7e724a2474bc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:09.871936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52251c-d5de-4d18-8f65-7e724a2474bc None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:09.874335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff52251c-d5de-4d18-8f65-7e724a2474bc None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:09.875793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff52251c-d5de-4d18-8f65-7e724a2474bc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:09.877792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 190/386] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:09 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:09.883351 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb042a-096f-4347-b501-85e06bf5a503 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:09.908957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3900eb3e-c67d-415a-8830-80a50aa0756d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:09.909259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3900eb3e-c67d-415a-8830-80a50aa0756d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:09.909522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3900eb3e-c67d-415a-8830-80a50aa0756d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:09.911275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 191/387] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:09.916657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8cd50-ce16-468e-b2f8-fc923788b118 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:09.916926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8cd50-ce16-468e-b2f8-fc923788b118 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:09.917189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda8cd50-ce16-468e-b2f8-fc923788b118 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:09.938950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fda8cd50-ce16-468e-b2f8-fc923788b118 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:09.940128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fda8cd50-ce16-468e-b2f8-fc923788b118 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:10.037675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:10.065141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb042a-096f-4347-b501-85e06bf5a503 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:10.075095 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 192/388] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:10.106574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116fde97-182b-438a-940d-1d91a03e5c17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:10.107247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116fde97-182b-438a-940d-1d91a03e5c17 None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:10.107527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116fde97-182b-438a-940d-1d91a03e5c17 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:10.107804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116fde97-182b-438a-940d-1d91a03e5c17 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26/groups/80da8e2dc5874812b1d79ec0374e05fd/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:10.109789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116fde97-182b-438a-940d-1d91a03e5c17 None admin] RBAC: Authorizing `identity:create_grant(project_id=8419cf9f31bb4753946ad654d60e9e26, group_id=80da8e2dc5874812b1d79ec0374e05fd, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:10.111239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116fde97-182b-438a-940d-1d91a03e5c17 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:10.183778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 197/389] 10.222.0.16 () {58 vars in 1442 bytes} [Thu Aug 25 04:53:09 2022] PUT /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26/groups/80da8e2dc5874812b1d79ec0374e05fd/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:10.401541 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda8cd50-ce16-468e-b2f8-fc923788b118 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:10.449200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:10.517202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda8cd50-ce16-468e-b2f8-fc923788b118 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:10.554304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e97ae30-0a5a-4ee6-af8b-8c971407b4ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:10.555260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e97ae30-0a5a-4ee6-af8b-8c971407b4ef None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:10.555730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e97ae30-0a5a-4ee6-af8b-8c971407b4ef None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:10.556260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e97ae30-0a5a-4ee6-af8b-8c971407b4ef None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:10.567261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e97ae30-0a5a-4ee6-af8b-8c971407b4ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-8e97ae30-0a5a-4ee6-af8b-8c971407b4ef None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:10.572277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:10.573719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:10.574521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:10.577038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 193/390] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:53:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:10.582855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116fde97-182b-438a-940d-1d91a03e5c17 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:10.704368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116fde97-182b-438a-940d-1d91a03e5c17 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:10.748174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:10.748854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:10.749140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:10.749412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:10.750107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:10.750415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:10.751796 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:10.767837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 198/391] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:10.773400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e97ae30-0a5a-4ee6-af8b-8c971407b4ef None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:10.918897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:10.922687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e97ae30-0a5a-4ee6-af8b-8c971407b4ef None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:10.961337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905c210a-4289-4af0-98a4-5e8481486f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:10.962181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c210a-4289-4af0-98a4-5e8481486f42 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:10.962469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c210a-4289-4af0-98a4-5e8481486f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:10.962741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905c210a-4289-4af0-98a4-5e8481486f42 None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:10.964386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905c210a-4289-4af0-98a4-5e8481486f42 None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:10.965856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905c210a-4289-4af0-98a4-5e8481486f42 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:10.967768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 194/392] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:10 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:10.973101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:10.980192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:11.106040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040e7b41-886b-4194-a11a-947a07d56bf6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:11.148186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd49c010-2578-49ad-958b-fdac8bb5d4f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:11.148830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd49c010-2578-49ad-958b-fdac8bb5d4f7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.149107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd49c010-2578-49ad-958b-fdac8bb5d4f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.149378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd49c010-2578-49ad-958b-fdac8bb5d4f7 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:11.154269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd49c010-2578-49ad-958b-fdac8bb5d4f7 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:11.155738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd49c010-2578-49ad-958b-fdac8bb5d4f7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:11.157738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 199/393] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:10 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:11.163184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905c210a-4289-4af0-98a4-5e8481486f42 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:11.296373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279376c-6888-4f46-8837-1a1c5661c066 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.297013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279376c-6888-4f46-8837-1a1c5661c066 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.297503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279376c-6888-4f46-8837-1a1c5661c066 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:11.299288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 200/394] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:11.303294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905c210a-4289-4af0-98a4-5e8481486f42 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:11.305026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da8464-887d-49ff-bf79-42394614df65 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.305519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da8464-887d-49ff-bf79-42394614df65 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.306032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40da8464-887d-49ff-bf79-42394614df65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:11.333973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-40da8464-887d-49ff-bf79-42394614df65 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:11.335752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-40da8464-887d-49ff-bf79-42394614df65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:11.354642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774d1a1-ad67-4156-878a-f5a8e72cd663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:11.354642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774d1a1-ad67-4156-878a-f5a8e72cd663 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.354642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774d1a1-ad67-4156-878a-f5a8e72cd663 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.354642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774d1a1-ad67-4156-878a-f5a8e72cd663 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26/groups/80da8e2dc5874812b1d79ec0374e05fd/roles/609ec30d147941b4a2b5508980aea7d0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:11.355031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774d1a1-ad67-4156-878a-f5a8e72cd663 None admin] RBAC: Authorizing `identity:create_grant(project_id=8419cf9f31bb4753946ad654d60e9e26, group_id=80da8e2dc5874812b1d79ec0374e05fd, role_id=609ec30d147941b4a2b5508980aea7d0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:11.355031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774d1a1-ad67-4156-878a-f5a8e72cd663 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:11.415752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:11.423132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 201/395] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:11.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 195/396] 10.222.0.16 () {58 vars in 1442 bytes} [Thu Aug 25 04:53:11 2022] PUT /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26/groups/80da8e2dc5874812b1d79ec0374e05fd/roles/609ec30d147941b4a2b5508980aea7d0 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:11.493606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5ca8bd-98d9-4edc-ba54-4e736e8ec72f None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.493831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5ca8bd-98d9-4edc-ba54-4e736e8ec72f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.494320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5ca8bd-98d9-4edc-ba54-4e736e8ec72f None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:11.496534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 202/397] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:11.502268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b0adf5-9ef6-4068-a57a-82acfc0ef923 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.502935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b0adf5-9ef6-4068-a57a-82acfc0ef923 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.503394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b0adf5-9ef6-4068-a57a-82acfc0ef923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:11.532174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e1b0adf5-9ef6-4068-a57a-82acfc0ef923 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:11.534352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e1b0adf5-9ef6-4068-a57a-82acfc0ef923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:11.563342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:11.635623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 196/398] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:11.751969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5ca8bd-98d9-4edc-ba54-4e736e8ec72f None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:11.805291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5a1517-c200-41f0-ac42-0e414d1d219d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.805572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5a1517-c200-41f0-ac42-0e414d1d219d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.806027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5a1517-c200-41f0-ac42-0e414d1d219d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:11.808561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 197/399] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:11.818790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e28983-5b9f-43f9-8f10-1c907d459857 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.819079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e28983-5b9f-43f9-8f10-1c907d459857 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.819356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e28983-5b9f-43f9-8f10-1c907d459857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:11.847188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-60e28983-5b9f-43f9-8f10-1c907d459857 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:11.848376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-60e28983-5b9f-43f9-8f10-1c907d459857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:11.927953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5ca8bd-98d9-4edc-ba54-4e736e8ec72f None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:11.980380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 198/400] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:11.981302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e28983-5b9f-43f9-8f10-1c907d459857 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:11.992064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3217d071-d1ee-4d22-8437-3d68a37cf94a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:11.992697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3217d071-d1ee-4d22-8437-3d68a37cf94a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:11.992970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3217d071-d1ee-4d22-8437-3d68a37cf94a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:11.993233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3217d071-d1ee-4d22-8437-3d68a37cf94a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.017962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3217d071-d1ee-4d22-8437-3d68a37cf94a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-3217d071-d1ee-4d22-8437-3d68a37cf94a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:12.023426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:12.024303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:12.025094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:12.029624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 203/401] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:12.032116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3217d071-d1ee-4d22-8437-3d68a37cf94a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.100717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:12.147290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:12.155672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3217d071-d1ee-4d22-8437-3d68a37cf94a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.174151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e28983-5b9f-43f9-8f10-1c907d459857 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.218237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.218849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.219129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.219397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.220057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:12.220357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:12.221749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:12.233608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 204/402] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:12.242520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59307c3-80ee-48f7-b93f-090049679d44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.243158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59307c3-80ee-48f7-b93f-090049679d44 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.243437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59307c3-80ee-48f7-b93f-090049679d44 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.243704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59307c3-80ee-48f7-b93f-090049679d44 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.246628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.266465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59307c3-80ee-48f7-b93f-090049679d44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d59307c3-80ee-48f7-b93f-090049679d44 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:12.271448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:12.272317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:12.273117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 199/403] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:53:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:12.280191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59307c3-80ee-48f7-b93f-090049679d44 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.393462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4adb386-65cd-4547-89de-da0c4e4a3945 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.394940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59307c3-80ee-48f7-b93f-090049679d44 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.417440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:12.429390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c935b9a-ca17-4cb0-ad13-7ac0221ba2c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.430155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c935b9a-ca17-4cb0-ad13-7ac0221ba2c7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.430487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c935b9a-ca17-4cb0-ad13-7ac0221ba2c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.430801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c935b9a-ca17-4cb0-ad13-7ac0221ba2c7 None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.432676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c935b9a-ca17-4cb0-ad13-7ac0221ba2c7 None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:12.434123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c935b9a-ca17-4cb0-ad13-7ac0221ba2c7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:12.436155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 205/404] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:12.436922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c935b9a-ca17-4cb0-ad13-7ac0221ba2c7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.438205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.439086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.439539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.440020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.440992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:12.441478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:12.443053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:12.453313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 200/405] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:12.454610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.467964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:12.554200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c935b9a-ca17-4cb0-ad13-7ac0221ba2c7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.567216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf381ac3-bcb8-42dc-8fc8-5c8d643cbf22 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.597107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a1225-e469-41e0-b24d-6d49069cce41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.597871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a1225-e469-41e0-b24d-6d49069cce41 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.598160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a1225-e469-41e0-b24d-6d49069cce41 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.598463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61a1225-e469-41e0-b24d-6d49069cce41 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.608707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa60fa7-b2cd-4f4d-9121-ea014e4b6dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.609613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa60fa7-b2cd-4f4d-9121-ea014e4b6dc3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.610109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa60fa7-b2cd-4f4d-9121-ea014e4b6dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.610579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa60fa7-b2cd-4f4d-9121-ea014e4b6dc3 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.611313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61a1225-e469-41e0-b24d-6d49069cce41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:12.612248 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa60fa7-b2cd-4f4d-9121-ea014e4b6dc3 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:12.613953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa60fa7-b2cd-4f4d-9121-ea014e4b6dc3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:12.615854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 201/406] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:12.617016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa60fa7-b2cd-4f4d-9121-ea014e4b6dc3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-a61a1225-e469-41e0-b24d-6d49069cce41 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:12.617133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:12.617994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:12.618804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:12.621834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 206/407] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:12.622660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a1225-e469-41e0-b24d-6d49069cce41 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.682145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:12.705560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:12.729470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa60fa7-b2cd-4f4d-9121-ea014e4b6dc3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.736100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61a1225-e469-41e0-b24d-6d49069cce41 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.771209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f267c4d-57c0-45fa-b7ea-65338c3dd42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.772121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f267c4d-57c0-45fa-b7ea-65338c3dd42e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.772817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f267c4d-57c0-45fa-b7ea-65338c3dd42e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.773277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f267c4d-57c0-45fa-b7ea-65338c3dd42e None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.774938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f267c4d-57c0-45fa-b7ea-65338c3dd42e None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:12.776843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f267c4d-57c0-45fa-b7ea-65338c3dd42e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:12.777428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107ac0b3-78d4-4c01-9022-8939bfa9d43e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.778230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107ac0b3-78d4-4c01-9022-8939bfa9d43e None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.778604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107ac0b3-78d4-4c01-9022-8939bfa9d43e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.778773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 202/408] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:12.779051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107ac0b3-78d4-4c01-9022-8939bfa9d43e None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/groups/80da8e2dc5874812b1d79ec0374e05fd/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.779597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f267c4d-57c0-45fa-b7ea-65338c3dd42e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.781146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107ac0b3-78d4-4c01-9022-8939bfa9d43e None admin] RBAC: Authorizing `identity:create_grant(project_id=2c8e2a16d7d34db6886ebfb9d8689b16, group_id=80da8e2dc5874812b1d79ec0374e05fd, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:12.782710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107ac0b3-78d4-4c01-9022-8939bfa9d43e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:12.895212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 207/409] 10.222.0.16 () {58 vars in 1442 bytes} [Thu Aug 25 04:53:12 2022] PUT /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/groups/80da8e2dc5874812b1d79ec0374e05fd/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:12.895440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f267c4d-57c0-45fa-b7ea-65338c3dd42e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:12.895522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107ac0b3-78d4-4c01-9022-8939bfa9d43e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.946548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789d598c-c7af-401a-abc5-6cb4cfc8116b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:12.947715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d598c-c7af-401a-abc5-6cb4cfc8116b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.947998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d598c-c7af-401a-abc5-6cb4cfc8116b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.948272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d598c-c7af-401a-abc5-6cb4cfc8116b None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.949444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789d598c-c7af-401a-abc5-6cb4cfc8116b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:12.950070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789d598c-c7af-401a-abc5-6cb4cfc8116b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:12.951835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789d598c-c7af-401a-abc5-6cb4cfc8116b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:12.969800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 203/410] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:12.971404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ddc65b-75ef-49a0-b678-2d4ae4e2d8f7 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:12.971717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ddc65b-75ef-49a0-b678-2d4ae4e2d8f7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:12.971999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ddc65b-75ef-49a0-b678-2d4ae4e2d8f7 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:12.973486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 204/411] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:12.978934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ddc65b-75ef-49a0-b678-2d4ae4e2d8f7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:12.991559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:13.045480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107ac0b3-78d4-4c01-9022-8939bfa9d43e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:13.060235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:13.085417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0364b9b4-5ca5-450e-b6e5-aae5e5b6e87f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:13.086550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364b9b4-5ca5-450e-b6e5-aae5e5b6e87f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.087382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364b9b4-5ca5-450e-b6e5-aae5e5b6e87f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.087986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364b9b4-5ca5-450e-b6e5-aae5e5b6e87f None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.090446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0364b9b4-5ca5-450e-b6e5-aae5e5b6e87f None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:13.092025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0364b9b4-5ca5-450e-b6e5-aae5e5b6e87f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:13.094346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 208/412] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:13.095557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc34f161-c1a2-4673-9e53-184880df8d63 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.096044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc34f161-c1a2-4673-9e53-184880df8d63 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.096396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc34f161-c1a2-4673-9e53-184880df8d63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.115436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ddc65b-75ef-49a0-b678-2d4ae4e2d8f7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:13.115681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-dc34f161-c1a2-4673-9e53-184880df8d63 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:13.116994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-dc34f161-c1a2-4673-9e53-184880df8d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:13.155232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f9715e-238d-411a-8dc0-29f985c385e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:13.156215 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f9715e-238d-411a-8dc0-29f985c385e0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.156636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f9715e-238d-411a-8dc0-29f985c385e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.157047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f9715e-238d-411a-8dc0-29f985c385e0 None admin] PATH_INFO: `/v3/groups/452fb2080a4c42eebf62abc27555bb2e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.158539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f9715e-238d-411a-8dc0-29f985c385e0 None admin] RBAC: Authorizing `identity:get_group(group_id=452fb2080a4c42eebf62abc27555bb2e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:13.160190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f9715e-238d-411a-8dc0-29f985c385e0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:13.162193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 205/413] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:12 2022] GET /identity/v3/groups/452fb2080a4c42eebf62abc27555bb2e => generated 229 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:13.163080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f9715e-238d-411a-8dc0-29f985c385e0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:13.203680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 209/414] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:13.204751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc34f161-c1a2-4673-9e53-184880df8d63 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:13.248171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:13.273879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f9715e-238d-411a-8dc0-29f985c385e0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:13.311288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2372418b-7387-4bc1-9dc0-143b22bef399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:13.312337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2372418b-7387-4bc1-9dc0-143b22bef399 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.313135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2372418b-7387-4bc1-9dc0-143b22bef399 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.313583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2372418b-7387-4bc1-9dc0-143b22bef399 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/groups/80da8e2dc5874812b1d79ec0374e05fd/roles/609ec30d147941b4a2b5508980aea7d0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.315980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2372418b-7387-4bc1-9dc0-143b22bef399 None admin] RBAC: Authorizing `identity:create_grant(project_id=2c8e2a16d7d34db6886ebfb9d8689b16, group_id=80da8e2dc5874812b1d79ec0374e05fd, role_id=609ec30d147941b4a2b5508980aea7d0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:13.318246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2372418b-7387-4bc1-9dc0-143b22bef399 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:13.380329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc34f161-c1a2-4673-9e53-184880df8d63 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:13.391255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:13.425196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 206/415] 10.222.0.16 () {58 vars in 1442 bytes} [Thu Aug 25 04:53:13 2022] PUT /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16/groups/80da8e2dc5874812b1d79ec0374e05fd/roles/609ec30d147941b4a2b5508980aea7d0 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:13.427182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c325b1-7d49-4cc8-a8c9-3250e2400a56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:13.428137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c325b1-7d49-4cc8-a8c9-3250e2400a56 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.428672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c325b1-7d49-4cc8-a8c9-3250e2400a56 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.429147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c325b1-7d49-4cc8-a8c9-3250e2400a56 None admin] PATH_INFO: `/v3/projects/f65e42175ca249f08b4d6dc3b72146f4` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.430771 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c325b1-7d49-4cc8-a8c9-3250e2400a56 None admin] RBAC: Authorizing `identity:get_project(project_id=f65e42175ca249f08b4d6dc3b72146f4)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:13.432751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c325b1-7d49-4cc8-a8c9-3250e2400a56 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:13.434973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 210/416] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:13 2022] GET /identity/v3/projects/f65e42175ca249f08b4d6dc3b72146f4 => generated 344 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:13.441810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2372418b-7387-4bc1-9dc0-143b22bef399 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:13.509684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c325b1-7d49-4cc8-a8c9-3250e2400a56 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:13.622578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2372418b-7387-4bc1-9dc0-143b22bef399 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:13.659913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c325b1-7d49-4cc8-a8c9-3250e2400a56 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:13.661457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88e0865-f2a8-4577-a528-f294daf1bbc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:13.663187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e0865-f2a8-4577-a528-f294daf1bbc4 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.663691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e0865-f2a8-4577-a528-f294daf1bbc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.664173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e0865-f2a8-4577-a528-f294daf1bbc4 None admin] PATH_INFO: `/v3/projects/f65e42175ca249f08b4d6dc3b72146f4/groups/452fb2080a4c42eebf62abc27555bb2e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.666380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88e0865-f2a8-4577-a528-f294daf1bbc4 None admin] RBAC: Authorizing `identity:create_grant(project_id=f65e42175ca249f08b4d6dc3b72146f4, group_id=452fb2080a4c42eebf62abc27555bb2e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:13.668711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88e0865-f2a8-4577-a528-f294daf1bbc4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:13.672189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:13.725685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e413e3-f556-4e04-b4c8-4a6aed1a54cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:13.725685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e413e3-f556-4e04-b4c8-4a6aed1a54cd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.725685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e413e3-f556-4e04-b4c8-4a6aed1a54cd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.725685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e413e3-f556-4e04-b4c8-4a6aed1a54cd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e413e3-f556-4e04-b4c8-4a6aed1a54cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-28e413e3-f556-4e04-b4c8-4a6aed1a54cd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:13.726239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:13.726976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:13.727847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 211/417] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:13.728601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d935b2-e360-4417-ac3e-47a4f6bd9344 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.728601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d935b2-e360-4417-ac3e-47a4f6bd9344 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.728601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d935b2-e360-4417-ac3e-47a4f6bd9344 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.728601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 212/418] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:13 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:13.728601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d935b2-e360-4417-ac3e-47a4f6bd9344 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:13.761757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:13.820535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 207/419] 10.222.0.16 () {58 vars in 1442 bytes} [Thu Aug 25 04:53:13 2022] PUT /identity/v3/projects/f65e42175ca249f08b4d6dc3b72146f4/groups/452fb2080a4c42eebf62abc27555bb2e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:13.821808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5c27d1-32e1-4701-a20f-67e4af333906 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.822130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5c27d1-32e1-4701-a20f-67e4af333906 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.822465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5c27d1-32e1-4701-a20f-67e4af333906 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.838623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d935b2-e360-4417-ac3e-47a4f6bd9344 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:13.844045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2e5c27d1-32e1-4701-a20f-67e4af333906 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:13.845412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2e5c27d1-32e1-4701-a20f-67e4af333906 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:13.888827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:13.889500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:13.889840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:13.890170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:13.890861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] RBAC: Adding query filter params (name=member) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:13.891203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:13.892643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:13.903465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 213/420] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:13.910130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:13.935651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 208/421] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:14.032554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:14.036416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840e5a0c-190d-46ef-b4d9-d35c02d3503d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:14.072318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff0e01-f3cf-4d69-a98b-beba30f1ceb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:14.073492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff0e01-f3cf-4d69-a98b-beba30f1ceb3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:14.073815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff0e01-f3cf-4d69-a98b-beba30f1ceb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:14.074146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff0e01-f3cf-4d69-a98b-beba30f1ceb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:14.144366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff0e01-f3cf-4d69-a98b-beba30f1ceb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:14.145993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff0e01-f3cf-4d69-a98b-beba30f1ceb3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:14.245041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:14.249869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 214/422] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 25 04:53:14.255786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5c27d1-32e1-4701-a20f-67e4af333906 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:14.261139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff0e01-f3cf-4d69-a98b-beba30f1ceb3 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:14.370082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff0e01-f3cf-4d69-a98b-beba30f1ceb3 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:14.392524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5c27d1-32e1-4701-a20f-67e4af333906 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:14.422853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:14.426700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fcaf6c-df64-4e83-b4eb-66202575b0a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:14.427357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fcaf6c-df64-4e83-b4eb-66202575b0a8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:14.427643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fcaf6c-df64-4e83-b4eb-66202575b0a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:14.427759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d5727-daf0-446e-9329-12bf80ccae91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:14.427938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fcaf6c-df64-4e83-b4eb-66202575b0a8 None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:14.428303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d5727-daf0-446e-9329-12bf80ccae91 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:14.428584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d5727-daf0-446e-9329-12bf80ccae91 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:14.428859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234d5727-daf0-446e-9329-12bf80ccae91 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:14.434367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fcaf6c-df64-4e83-b4eb-66202575b0a8 None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:14.435862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fcaf6c-df64-4e83-b4eb-66202575b0a8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:14.437777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 215/423] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:14 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:14.440433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234d5727-daf0-446e-9329-12bf80ccae91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:14.443673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fcaf6c-df64-4e83-b4eb-66202575b0a8 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-234d5727-daf0-446e-9329-12bf80ccae91 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:14.445389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:14.446293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:14.447240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 209/424] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:53:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:14.453961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d5727-daf0-446e-9329-12bf80ccae91 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:14.580214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234d5727-daf0-446e-9329-12bf80ccae91 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:14.581815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:14.613792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fcaf6c-df64-4e83-b4eb-66202575b0a8 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:14.618460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:14.619079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:14.619388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:14.619663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:14.620389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:14.620633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:14.622035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:14.630697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 210/425] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:14.639044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:14.660110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543eebea-412f-4b29-9d34-d057509aa36f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:14.660806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543eebea-412f-4b29-9d34-d057509aa36f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:14.661130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543eebea-412f-4b29-9d34-d057509aa36f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:14.661415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543eebea-412f-4b29-9d34-d057509aa36f None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:14.663396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543eebea-412f-4b29-9d34-d057509aa36f None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:14.665137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543eebea-412f-4b29-9d34-d057509aa36f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:14.668239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 216/426] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:14 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:14.674079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543eebea-412f-4b29-9d34-d057509aa36f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:14.741936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:14.806916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f6220a-4698-44e9-bd72-0eacfd96dcc5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:14.845713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91555635-afb1-46ff-9689-3d43531bfaf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:14.846443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91555635-afb1-46ff-9689-3d43531bfaf6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:14.846735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91555635-afb1-46ff-9689-3d43531bfaf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:14.847012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91555635-afb1-46ff-9689-3d43531bfaf6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:14.852332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543eebea-412f-4b29-9d34-d057509aa36f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:14.887781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:14.888394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:14.888676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:14.888953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:14.889773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] RBAC: Adding query filter params (group.id=80da8e2dc5874812b1d79ec0374e05fd, role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=8419cf9f31bb4753946ad654d60e9e26) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:14.890092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:14.891591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:14.893416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:14.906445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 217/427] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:14 2022] GET /identity/v3/role_assignments?group.id=80da8e2dc5874812b1d79ec0374e05fd&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 611 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:14.920795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91555635-afb1-46ff-9689-3d43531bfaf6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:14.922660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91555635-afb1-46ff-9689-3d43531bfaf6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:14.973139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:15.032590 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 211/428] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:15.038628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:15.177438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c57e62-0161-4093-ad81-74429925dc89 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:15.182804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176a260c-0a77-4b79-992a-9065c1f854ee None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.183247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176a260c-0a77-4b79-992a-9065c1f854ee None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.183637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176a260c-0a77-4b79-992a-9065c1f854ee None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.185436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 212/429] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:15.191528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7224527a-5b85-4d41-94a9-cd7ca628a012 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.192071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7224527a-5b85-4d41-94a9-cd7ca628a012 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.192540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7224527a-5b85-4d41-94a9-cd7ca628a012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.218938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd8e0d7-a99b-4e7e-8316-8060da906a5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:15.219169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8e0d7-a99b-4e7e-8316-8060da906a5b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.219401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8e0d7-a99b-4e7e-8316-8060da906a5b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.219679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd8e0d7-a99b-4e7e-8316-8060da906a5b None admin] PATH_INFO: `/v3/projects/8419cf9f31bb4753946ad654d60e9e26` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.222823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd8e0d7-a99b-4e7e-8316-8060da906a5b None admin] RBAC: Authorizing `identity:get_project(project_id=8419cf9f31bb4753946ad654d60e9e26)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:15.222823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd8e0d7-a99b-4e7e-8316-8060da906a5b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:15.226386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 218/430] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:15 2022] GET /identity/v3/projects/8419cf9f31bb4753946ad654d60e9e26 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:15.229897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd8e0d7-a99b-4e7e-8316-8060da906a5b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:15.233090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7224527a-5b85-4d41-94a9-cd7ca628a012 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:15.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7224527a-5b85-4d41-94a9-cd7ca628a012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:15.303008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:15.314577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 213/431] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:15.342874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd8e0d7-a99b-4e7e-8316-8060da906a5b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:15.383391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0629738-dec5-4809-97fe-07195c0d1dbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:15.384030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0629738-dec5-4809-97fe-07195c0d1dbb None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.384310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0629738-dec5-4809-97fe-07195c0d1dbb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.384584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0629738-dec5-4809-97fe-07195c0d1dbb None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.386491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0629738-dec5-4809-97fe-07195c0d1dbb None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:15.387955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0629738-dec5-4809-97fe-07195c0d1dbb None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:15.389881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 219/432] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:15 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:15.395512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7224527a-5b85-4d41-94a9-cd7ca628a012 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:15.491175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:15.515180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7224527a-5b85-4d41-94a9-cd7ca628a012 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:15.554957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:15.555564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.555841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.556112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.556901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] RBAC: Adding query filter params (group.id=80da8e2dc5874812b1d79ec0374e05fd, role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=8419cf9f31bb4753946ad654d60e9e26) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:15.557211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:15.558658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:15.564814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 214/433] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:15 2022] GET /identity/v3/role_assignments?group.id=80da8e2dc5874812b1d79ec0374e05fd&scope.project.id=8419cf9f31bb4753946ad654d60e9e26&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 611 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:15.612693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0629738-dec5-4809-97fe-07195c0d1dbb None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:15.729135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0629738-dec5-4809-97fe-07195c0d1dbb None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:15.768267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2560dc-d31c-4fb8-ba31-80274d6a72ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:15.768830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2560dc-d31c-4fb8-ba31-80274d6a72ba None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.769109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2560dc-d31c-4fb8-ba31-80274d6a72ba None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.769377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2560dc-d31c-4fb8-ba31-80274d6a72ba None admin] PATH_INFO: `/v3/roles/member` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.771286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:15.782860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2560dc-d31c-4fb8-ba31-80274d6a72ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-aa2560dc-d31c-4fb8-ba31-80274d6a72ba None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:15.787881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:15.794575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:15.795486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 220/434] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:15.796152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:15.855248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f27b9fc-7d21-427b-81e8-a72fa8ad8434 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.855740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f27b9fc-7d21-427b-81e8-a72fa8ad8434 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.855813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f27b9fc-7d21-427b-81e8-a72fa8ad8434 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.857126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 221/435] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:15.862674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad41c260-1f0e-44d7-a7ab-bddb3becab25 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.862961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad41c260-1f0e-44d7-a7ab-bddb3becab25 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.863237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad41c260-1f0e-44d7-a7ab-bddb3becab25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.886622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ad41c260-1f0e-44d7-a7ab-bddb3becab25 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:15.887985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ad41c260-1f0e-44d7-a7ab-bddb3becab25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:15.914773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb1a88b-7383-4a60-a0e4-2ae48564fa46 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:15.959844 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:15.962304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:15.966787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:15.966787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:15.966787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] RBAC: Adding query filter params (name=member) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:15.966787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:15.966787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:15.973704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 222/436] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:15.977115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 215/437] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:53:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:15.983108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad41c260-1f0e-44d7-a7ab-bddb3becab25 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.093466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad41c260-1f0e-44d7-a7ab-bddb3becab25 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:16.116777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:16.128112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034fd13-b2cf-4de3-ae67-993d476aeb22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:16.128728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034fd13-b2cf-4de3-ae67-993d476aeb22 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.129011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034fd13-b2cf-4de3-ae67-993d476aeb22 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.129285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0034fd13-b2cf-4de3-ae67-993d476aeb22 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.197961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0034fd13-b2cf-4de3-ae67-993d476aeb22 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:16.199499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0034fd13-b2cf-4de3-ae67-993d476aeb22 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:16.252251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.255138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:16.311789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 223/438] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 25 04:53:16.317372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034fd13-b2cf-4de3-ae67-993d476aeb22 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.344357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:16.358125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61abc44-c850-4131-9adb-f66ffea03d69 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:16.396679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f87d92b-80cf-47d9-94ce-7bc0c1b71896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:16.397334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87d92b-80cf-47d9-94ce-7bc0c1b71896 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.397638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87d92b-80cf-47d9-94ce-7bc0c1b71896 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87d92b-80cf-47d9-94ce-7bc0c1b71896 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.408629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f87d92b-80cf-47d9-94ce-7bc0c1b71896 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-8f87d92b-80cf-47d9-94ce-7bc0c1b71896 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:16.413849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:16.414766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:16.415578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 216/439] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:53:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:16.417711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c59118-c486-494c-a35c-f34057cfb89b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.417991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c59118-c486-494c-a35c-f34057cfb89b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.418262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c59118-c486-494c-a35c-f34057cfb89b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.419860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 217/440] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:16.420562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c59118-c486-494c-a35c-f34057cfb89b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.426251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0034fd13-b2cf-4de3-ae67-993d476aeb22 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:16.459773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f56060-6dfc-4c71-8c34-c0f5d86cf419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:16.460614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f56060-6dfc-4c71-8c34-c0f5d86cf419 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.460888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f56060-6dfc-4c71-8c34-c0f5d86cf419 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.461155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f56060-6dfc-4c71-8c34-c0f5d86cf419 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.462545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f56060-6dfc-4c71-8c34-c0f5d86cf419 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:16.464583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f56060-6dfc-4c71-8c34-c0f5d86cf419 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:16.466412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 224/441] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:16 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:16.467310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a041b4-c6f2-4e1b-9240-fe16751e0fb7 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.467592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a041b4-c6f2-4e1b-9240-fe16751e0fb7 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.467860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a041b4-c6f2-4e1b-9240-fe16751e0fb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.484580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-56a041b4-c6f2-4e1b-9240-fe16751e0fb7 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:16.485067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:16.485918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-56a041b4-c6f2-4e1b-9240-fe16751e0fb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:16.524034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c59118-c486-494c-a35c-f34057cfb89b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:16.551626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 225/442] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:16.552455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a041b4-c6f2-4e1b-9240-fe16751e0fb7 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.564450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:16.565056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.565333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.565617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.566308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:16.566609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:16.567974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:16.576362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 218/443] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:16.577098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:16.581549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.656155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a041b4-c6f2-4e1b-9240-fe16751e0fb7 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:16.689860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f88062-3bd9-4883-8908-b6985b265dce None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:16.690942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7710e18b-7d2f-4a73-a803-b3bf470f7d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:16.691574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7710e18b-7d2f-4a73-a803-b3bf470f7d07 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.691852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7710e18b-7d2f-4a73-a803-b3bf470f7d07 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.692127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7710e18b-7d2f-4a73-a803-b3bf470f7d07 None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.693222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7710e18b-7d2f-4a73-a803-b3bf470f7d07 None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:16.694676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7710e18b-7d2f-4a73-a803-b3bf470f7d07 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:16.700908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 226/444] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:16 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:16.701907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7710e18b-7d2f-4a73-a803-b3bf470f7d07 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.714924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:16.733118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb08981-cc23-45b5-b004-008fe64af477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:16.733722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb08981-cc23-45b5-b004-008fe64af477 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.734006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb08981-cc23-45b5-b004-008fe64af477 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.734281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb08981-cc23-45b5-b004-008fe64af477 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.804147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb08981-cc23-45b5-b004-008fe64af477 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:16.804791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:16.805861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb08981-cc23-45b5-b004-008fe64af477 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:16.816634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7710e18b-7d2f-4a73-a803-b3bf470f7d07 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:16.852070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:16.852675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:16.852955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:16.853228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:16.854013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] RBAC: Adding query filter params (group.id=80da8e2dc5874812b1d79ec0374e05fd, role.id=0940d9778fb041fbaf7c4f4317feb2d5, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:16.854327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:16.855746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:16.861425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 227/445] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:16 2022] GET /identity/v3/role_assignments?group.id=80da8e2dc5874812b1d79ec0374e05fd&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=0940d9778fb041fbaf7c4f4317feb2d5 => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:16.862216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.920522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 219/446] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 25 04:53:16.926438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb08981-cc23-45b5-b004-008fe64af477 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:16.954350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:16.974027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1741d059-ae23-4aad-b813-b5e84311eaef None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.014238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66af2f87-d6da-4876-91f8-1a054d847176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.014879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66af2f87-d6da-4876-91f8-1a054d847176 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.015164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66af2f87-d6da-4876-91f8-1a054d847176 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.015439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66af2f87-d6da-4876-91f8-1a054d847176 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.029102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66af2f87-d6da-4876-91f8-1a054d847176 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-66af2f87-d6da-4876-91f8-1a054d847176 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:17.034367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:17.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:17.036089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:17.036800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 228/447] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:17.042337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66af2f87-d6da-4876-91f8-1a054d847176 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:17.044453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb08981-cc23-45b5-b004-008fe64af477 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.062152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:17.085651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d07a11-6b1d-498a-ae4a-b0ec7b7872c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.086148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d07a11-6b1d-498a-ae4a-b0ec7b7872c8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.086423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d07a11-6b1d-498a-ae4a-b0ec7b7872c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.086692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d07a11-6b1d-498a-ae4a-b0ec7b7872c8 None admin] PATH_INFO: `/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.088143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d07a11-6b1d-498a-ae4a-b0ec7b7872c8 None admin] RBAC: Authorizing `identity:get_project(project_id=2c8e2a16d7d34db6886ebfb9d8689b16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:17.092219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d07a11-6b1d-498a-ae4a-b0ec7b7872c8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:17.092219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 220/448] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:16 2022] GET /identity/v3/projects/2c8e2a16d7d34db6886ebfb9d8689b16 => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:17.096722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d07a11-6b1d-498a-ae4a-b0ec7b7872c8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:17.168225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66af2f87-d6da-4876-91f8-1a054d847176 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.207291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.207826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.208111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.208390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.209051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:17.209361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:17.209781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:17.210853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:17.219639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d07a11-6b1d-498a-ae4a-b0ec7b7872c8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.226707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 229/449] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:17.227851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:17.254199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e5060-fb44-48cf-9c2a-2378afd3dfcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.254818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e5060-fb44-48cf-9c2a-2378afd3dfcb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.255095 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e5060-fb44-48cf-9c2a-2378afd3dfcb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.255365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e5060-fb44-48cf-9c2a-2378afd3dfcb None admin] PATH_INFO: `/v3/groups/80da8e2dc5874812b1d79ec0374e05fd` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.256468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e5060-fb44-48cf-9c2a-2378afd3dfcb None admin] RBAC: Authorizing `identity:get_group(group_id=80da8e2dc5874812b1d79ec0374e05fd)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:17.257916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e5060-fb44-48cf-9c2a-2378afd3dfcb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:17.259465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 221/450] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:17 2022] GET /identity/v3/groups/80da8e2dc5874812b1d79ec0374e05fd => generated 226 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:17.265703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e5060-fb44-48cf-9c2a-2378afd3dfcb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:17.304344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:17.326560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c3bc94-e58f-428e-9c17-2ad120a7b997 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.361739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a9e0f9-d0cd-4ed7-80a8-2f1ff3d079db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.362225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e5060-fb44-48cf-9c2a-2378afd3dfcb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.362335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9e0f9-d0cd-4ed7-80a8-2f1ff3d079db None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.362614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9e0f9-d0cd-4ed7-80a8-2f1ff3d079db None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.362887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9e0f9-d0cd-4ed7-80a8-2f1ff3d079db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.395662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.396262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.396540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.396808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.397524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] RBAC: Adding query filter params (group.id=80da8e2dc5874812b1d79ec0374e05fd, role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:17.397858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:17.399285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:17.405400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 222/451] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:17 2022] GET /identity/v3/role_assignments?group.id=80da8e2dc5874812b1d79ec0374e05fd&scope.project.id=2c8e2a16d7d34db6886ebfb9d8689b16&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 611 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:17.434883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a9e0f9-d0cd-4ed7-80a8-2f1ff3d079db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:17.436367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a9e0f9-d0cd-4ed7-80a8-2f1ff3d079db None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:17.514769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:17.538164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 230/452] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 25 04:53:17.544058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:17.571480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:17.647157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f6fb31-71d2-45a1-a432-c00dc713356e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.680984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8d070c-7ae7-4039-95f1-4c5a6fb16547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.682130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8d070c-7ae7-4039-95f1-4c5a6fb16547 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.682413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8d070c-7ae7-4039-95f1-4c5a6fb16547 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.682690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8d070c-7ae7-4039-95f1-4c5a6fb16547 None admin] PATH_INFO: `/v3/projects/f65e42175ca249f08b4d6dc3b72146f4` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.683810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8d070c-7ae7-4039-95f1-4c5a6fb16547 None admin] RBAC: Authorizing `identity:get_project(project_id=f65e42175ca249f08b4d6dc3b72146f4)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:17.685241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8d070c-7ae7-4039-95f1-4c5a6fb16547 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:17.687074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 223/453] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:53:17 2022] GET /identity/v3/projects/f65e42175ca249f08b4d6dc3b72146f4 => generated 344 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:17.692386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a9e0f9-d0cd-4ed7-80a8-2f1ff3d079db None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:17.794488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a9e0f9-d0cd-4ed7-80a8-2f1ff3d079db None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.826888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67de7a09-9344-4a09-a9b1-e2287c5e6007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.827507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de7a09-9344-4a09-a9b1-e2287c5e6007 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.827784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de7a09-9344-4a09-a9b1-e2287c5e6007 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.828062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67de7a09-9344-4a09-a9b1-e2287c5e6007 None admin] PATH_INFO: `/v3/groups/452fb2080a4c42eebf62abc27555bb2e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.829148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67de7a09-9344-4a09-a9b1-e2287c5e6007 None admin] RBAC: Authorizing `identity:get_group(group_id=452fb2080a4c42eebf62abc27555bb2e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:17.830624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67de7a09-9344-4a09-a9b1-e2287c5e6007 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:17.832192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 231/454] 10.222.0.16 () {56 vars in 1180 bytes} [Thu Aug 25 04:53:17 2022] GET /identity/v3/groups/452fb2080a4c42eebf62abc27555bb2e => generated 229 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:17.838379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8d070c-7ae7-4039-95f1-4c5a6fb16547 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:17.927036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:17.935897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8d070c-7ae7-4039-95f1-4c5a6fb16547 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:17.968603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0962616-69a1-4508-9415-033d90bae381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:17.969194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0962616-69a1-4508-9415-033d90bae381 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:17.969473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0962616-69a1-4508-9415-033d90bae381 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:17.969775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0962616-69a1-4508-9415-033d90bae381 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:17.970491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0962616-69a1-4508-9415-033d90bae381 None admin] RBAC: Adding query filter params (group.id=452fb2080a4c42eebf62abc27555bb2e, role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=f65e42175ca249f08b4d6dc3b72146f4) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:17.970802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0962616-69a1-4508-9415-033d90bae381 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:17.972197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0962616-69a1-4508-9415-033d90bae381 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:17.977853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 224/455] 10.222.0.16 () {56 vars in 1376 bytes} [Thu Aug 25 04:53:17 2022] GET /identity/v3/role_assignments?group.id=452fb2080a4c42eebf62abc27555bb2e&scope.project.id=f65e42175ca249f08b4d6dc3b72146f4&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 611 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:20.191140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b386af3-1ff8-4002-a3cc-31da86f8f79f None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:20.191655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b386af3-1ff8-4002-a3cc-31da86f8f79f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:20.191655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b386af3-1ff8-4002-a3cc-31da86f8f79f None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:20.193290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 232/456] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:20.198660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62453d5-add4-485c-9f7c-4d305bd3cf88 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:20.199042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62453d5-add4-485c-9f7c-4d305bd3cf88 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:20.199225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62453d5-add4-485c-9f7c-4d305bd3cf88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:20.232974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e62453d5-add4-485c-9f7c-4d305bd3cf88 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:20.234453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e62453d5-add4-485c-9f7c-4d305bd3cf88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:20.331667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 225/457] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:20.489403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdcab3f-3bf3-4a9d-9ab9-10a6829d0aa4 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:20.490024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdcab3f-3bf3-4a9d-9ab9-10a6829d0aa4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:20.490496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdcab3f-3bf3-4a9d-9ab9-10a6829d0aa4 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:20.492260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 233/458] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:20.498640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cece395-3af5-44c0-b820-8e7abeac365a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:20.499142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cece395-3af5-44c0-b820-8e7abeac365a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:20.499565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cece395-3af5-44c0-b820-8e7abeac365a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:20.522712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2cece395-3af5-44c0-b820-8e7abeac365a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:20.524246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2cece395-3af5-44c0-b820-8e7abeac365a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:20.614373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 226/459] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:20.698590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdcab3f-3bf3-4a9d-9ab9-10a6829d0aa4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:20.759290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:20.838419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdcab3f-3bf3-4a9d-9ab9-10a6829d0aa4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:20.881472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:20.882336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:20.882784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:20.883203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:20.897157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:20.898817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:20.918346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:20.919493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:20.920547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:20.920547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:20.920547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:20.920985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cece395-3af5-44c0-b820-8e7abeac365a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:20.922087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 234/460] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:20.935594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:20.944125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:21.061576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cece395-3af5-44c0-b820-8e7abeac365a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:21.081434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:21.088869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb5e663-f1b8-4332-8805-08820910fa0f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:21.105992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f07d68-abc4-4c5f-ad50-83231c3b69ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:21.107240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f07d68-abc4-4c5f-ad50-83231c3b69ea None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.107796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f07d68-abc4-4c5f-ad50-83231c3b69ea None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.108303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f07d68-abc4-4c5f-ad50-83231c3b69ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.121408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f07d68-abc4-4c5f-ad50-83231c3b69ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:21.123803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f07d68-abc4-4c5f-ad50-83231c3b69ea None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:21.130231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d7f220-7610-4ebb-8c0c-75b1e40d1ebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:21.130949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d7f220-7610-4ebb-8c0c-75b1e40d1ebd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.131234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d7f220-7610-4ebb-8c0c-75b1e40d1ebd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.131507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d7f220-7610-4ebb-8c0c-75b1e40d1ebd None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.132205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d7f220-7610-4ebb-8c0c-75b1e40d1ebd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:21.132519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d7f220-7610-4ebb-8c0c-75b1e40d1ebd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:21.133915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d7f220-7610-4ebb-8c0c-75b1e40d1ebd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e6f07d68-abc4-4c5f-ad50-83231c3b69ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:21.143076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:21.144170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:21.145316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:21.145316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:21.145316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:21.147513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 227/461] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:21.148831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040b6dc-3082-4aba-9003-0fff29c4869d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.149250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040b6dc-3082-4aba-9003-0fff29c4869d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.149685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d040b6dc-3082-4aba-9003-0fff29c4869d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.151804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 228/462] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:21.158621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d040b6dc-3082-4aba-9003-0fff29c4869d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:21.165317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 235/463] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:21.166959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f426ecd-c72d-4e4d-a67b-2646d33ec1f5 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.167460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f426ecd-c72d-4e4d-a67b-2646d33ec1f5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.167985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f426ecd-c72d-4e4d-a67b-2646d33ec1f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.186490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2f426ecd-c72d-4e4d-a67b-2646d33ec1f5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:21.187661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2f426ecd-c72d-4e4d-a67b-2646d33ec1f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:21.242467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:21.261775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 236/464] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:21.270076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f426ecd-c72d-4e4d-a67b-2646d33ec1f5 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:21.288630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d040b6dc-3082-4aba-9003-0fff29c4869d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:21.327016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed6a7c3-f3b3-4022-85f2-5bde5d74f81c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:21.327548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed6a7c3-f3b3-4022-85f2-5bde5d74f81c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.327829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed6a7c3-f3b3-4022-85f2-5bde5d74f81c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.328097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed6a7c3-f3b3-4022-85f2-5bde5d74f81c None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.328796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed6a7c3-f3b3-4022-85f2-5bde5d74f81c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:21.329103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed6a7c3-f3b3-4022-85f2-5bde5d74f81c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:21.330934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed6a7c3-f3b3-4022-85f2-5bde5d74f81c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:21.349432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 229/465] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:21.350862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221f42dd-1516-4b05-8946-382bba13d962 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.351150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221f42dd-1516-4b05-8946-382bba13d962 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.351426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221f42dd-1516-4b05-8946-382bba13d962 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.357218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 230/466] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:21.357218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221f42dd-1516-4b05-8946-382bba13d962 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:21.375784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:21.392864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f426ecd-c72d-4e4d-a67b-2646d33ec1f5 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:21.434558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02f0de8-f81f-4cb8-9cff-9eebc623cf4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:21.435451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f0de8-f81f-4cb8-9cff-9eebc623cf4f None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.435898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f0de8-f81f-4cb8-9cff-9eebc623cf4f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.436431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02f0de8-f81f-4cb8-9cff-9eebc623cf4f None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.437316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02f0de8-f81f-4cb8-9cff-9eebc623cf4f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:21.439036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02f0de8-f81f-4cb8-9cff-9eebc623cf4f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:21.475268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221f42dd-1516-4b05-8946-382bba13d962 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:21.503606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:21.512295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8820fac-df08-4efe-8105-443c3e64d7f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:21.512914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8820fac-df08-4efe-8105-443c3e64d7f4 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.513183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8820fac-df08-4efe-8105-443c3e64d7f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.513452 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8820fac-df08-4efe-8105-443c3e64d7f4 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.514118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8820fac-df08-4efe-8105-443c3e64d7f4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:21.515541 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8820fac-df08-4efe-8105-443c3e64d7f4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:21.532802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 237/467] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:53:21 2022] POST /identity/v3/users => generated 246 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:21.534362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a581c8b-41bf-409e-b615-636015eaa6df None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.534892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a581c8b-41bf-409e-b615-636015eaa6df None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.535362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a581c8b-41bf-409e-b615-636015eaa6df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.557794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1a581c8b-41bf-409e-b615-636015eaa6df None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:21.559619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1a581c8b-41bf-409e-b615-636015eaa6df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:21.607226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 231/468] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:53:21 2022] POST /identity/v3/users => generated 248 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:21.607837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8820fac-df08-4efe-8105-443c3e64d7f4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:21.642710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 238/469] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:21.644512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dbdef2-1416-4d4d-92db-1927472f2bd9 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.645087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dbdef2-1416-4d4d-92db-1927472f2bd9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.645620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dbdef2-1416-4d4d-92db-1927472f2bd9 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.647705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 239/470] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:21.654142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a908a-bc57-40e4-a649-19cc389eaddc None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.654475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a908a-bc57-40e4-a649-19cc389eaddc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.654736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a908a-bc57-40e4-a649-19cc389eaddc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.682874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4a5a908a-bc57-40e4-a649-19cc389eaddc None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:21.684192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4a5a908a-bc57-40e4-a649-19cc389eaddc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:21.777210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8820fac-df08-4efe-8105-443c3e64d7f4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:21.805722 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 240/471] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:21.819751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:21.820658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:21.821135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:21.821620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:21.833165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:21.834870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:21.850427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:21.851338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:21.852193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:21.852193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:21.852193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:21.852193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 232/472] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:21.859309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a908a-bc57-40e4-a649-19cc389eaddc None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:21.861873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:21.943553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:21.975902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a908a-bc57-40e4-a649-19cc389eaddc None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:21.988956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:22.021669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:22.022839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:22.023318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:22.023834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:22.024845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:22.026450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:22.028351 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:22.044912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 241/473] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:22.050392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:22.071502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b195caa4-81ba-4910-abdd-ca92a7e00eb3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:22.128376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:22.132165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:22.132802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:22.133115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:22.133348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:22.143866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:22.145326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:22.161105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:22.162050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:22.162896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:22.162896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:22.162896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:22.162896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 233/474] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:22.163685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:22.169355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4a830c-dc50-4425-9976-a22b83b007b0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:22.206133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bf448c-9b14-482f-810d-fd5d1429a1e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:22.206873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf448c-9b14-482f-810d-fd5d1429a1e9 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:22.207148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf448c-9b14-482f-810d-fd5d1429a1e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:22.207415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bf448c-9b14-482f-810d-fd5d1429a1e9 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:22.208048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bf448c-9b14-482f-810d-fd5d1429a1e9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:22.209483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bf448c-9b14-482f-810d-fd5d1429a1e9 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:22.266250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:22.279831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce2d4ef-605f-468a-b702-2fbb60741bad None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:22.305849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 242/475] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:53:22 2022] POST /identity/v3/users => generated 248 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:22.307215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bf448c-9b14-482f-810d-fd5d1429a1e9 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:22.339478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:22.341252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:22.341540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:22.341840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:22.342682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:22.347759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:22.416352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.sql.core [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 25 04:53:22.416352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 25 04:53:22.416352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [parameters: {'id': 'f05cc506bfbb40689318f718490539a3', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 25 04:53:22.416352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81762) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result = self._query(query) Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application conn.query(q) Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result.read() Aug 25 04:53:22.437173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return ref.to_dict() Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 25 04:53:22.438165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application next(self.gen) Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application yield resource Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application next(self.gen) Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.session.rollback() Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.force_reraise() Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise self.value Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application session.commit() Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 25 04:53:22.439100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._prepare_impl() Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.session.flush() Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._flush(objects) Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application compat.raise_( Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application flush_context.execute() Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rec.execute(self) Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 25 04:53:22.439982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 25 04:53:22.441518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result = self._query(query) Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application conn.query(q) Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result.read() Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 25 04:53:22.442367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [parameters: {'id': 'f05cc506bfbb40689318f718490539a3', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:22.443167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 25 04:53:22.443987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:22.444829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 234/476] 10.222.0.16 () {60 vars in 1131 bytes} [Thu Aug 25 04:53:22 2022] POST /identity/v3/roles => generated 144 bytes in 275 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:22.454388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:22.483826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bf448c-9b14-482f-810d-fd5d1429a1e9 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:22.503272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:22.522071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:22.522700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:22.522968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:22.523242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:22.523880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:22.524180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:22.525794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:22.545216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 243/477] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:22.548939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:22.586307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95705908-e03c-4e05-9007-1840e3524d28 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:22.624646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109c6cd5-f0d3-4a1e-92a3-71c8ee5c6628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:22.625636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109c6cd5-f0d3-4a1e-92a3-71c8ee5c6628 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:22.626109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109c6cd5-f0d3-4a1e-92a3-71c8ee5c6628 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:22.626594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109c6cd5-f0d3-4a1e-92a3-71c8ee5c6628 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:22.638155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109c6cd5-f0d3-4a1e-92a3-71c8ee5c6628 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-109c6cd5-f0d3-4a1e-92a3-71c8ee5c6628 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:22.643303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:22.644188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 25 04:53:22.644986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:22.647630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 235/478] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:53:22 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:22.653462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109c6cd5-f0d3-4a1e-92a3-71c8ee5c6628 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:22.654621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bd202d-24a9-495b-9f4d-844f4197a342 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:22.674177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:22.690499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7591315-489e-4fcd-ac7f-bed9ec0fa7d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:22.691028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7591315-489e-4fcd-ac7f-bed9ec0fa7d1 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:22.691310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7591315-489e-4fcd-ac7f-bed9ec0fa7d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:22.691584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7591315-489e-4fcd-ac7f-bed9ec0fa7d1 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:22.692226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7591315-489e-4fcd-ac7f-bed9ec0fa7d1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:22.693694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7591315-489e-4fcd-ac7f-bed9ec0fa7d1 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:22.772933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109c6cd5-f0d3-4a1e-92a3-71c8ee5c6628 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:22.774912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 244/479] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:53:22 2022] POST /identity/v3/users => generated 249 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:22.816100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7959fc-dfe5-4708-99ad-3cddd1ceea8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:22.817174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7959fc-dfe5-4708-99ad-3cddd1ceea8b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:22.817672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7959fc-dfe5-4708-99ad-3cddd1ceea8b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:22.818138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7959fc-dfe5-4708-99ad-3cddd1ceea8b None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:22.819043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7959fc-dfe5-4708-99ad-3cddd1ceea8b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:22.819560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7959fc-dfe5-4708-99ad-3cddd1ceea8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:22.821154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7959fc-dfe5-4708-99ad-3cddd1ceea8b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:22.833132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 236/480] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:23.981487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae308090-0be3-429a-af8d-c92104dfee7a None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:23.982030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae308090-0be3-429a-af8d-c92104dfee7a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:23.982107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae308090-0be3-429a-af8d-c92104dfee7a None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:23.983670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 245/481] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:23.999042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941d987-700e-4716-9647-a866be21ea23 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:23.999866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941d987-700e-4716-9647-a866be21ea23 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:24.000367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941d987-700e-4716-9647-a866be21ea23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:24.026534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e941d987-700e-4716-9647-a866be21ea23 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:24.028026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e941d987-700e-4716-9647-a866be21ea23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:24.077329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3088f908-0a78-4e78-b656-ab2abf5abb02 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:24.077651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3088f908-0a78-4e78-b656-ab2abf5abb02 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:24.077912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3088f908-0a78-4e78-b656-ab2abf5abb02 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:24.079487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 246/482] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:24.091706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d075030-6de7-41d1-92ea-59818898338e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:24.091992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d075030-6de7-41d1-92ea-59818898338e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:24.092270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d075030-6de7-41d1-92ea-59818898338e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:24.114311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1d075030-6de7-41d1-92ea-59818898338e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:24.115505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1d075030-6de7-41d1-92ea-59818898338e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:24.131806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 237/483] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:24.166311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:24.193023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 247/484] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:24.495765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e941d987-700e-4716-9647-a866be21ea23 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:24.499408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:24.540146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d075030-6de7-41d1-92ea-59818898338e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:24.600443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e941d987-700e-4716-9647-a866be21ea23 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:24.636491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff346ae-399e-49d8-87af-3c809cfc7dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:24.637148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff346ae-399e-49d8-87af-3c809cfc7dba None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:24.637553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff346ae-399e-49d8-87af-3c809cfc7dba None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:24.637852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff346ae-399e-49d8-87af-3c809cfc7dba None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:24.648416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff346ae-399e-49d8-87af-3c809cfc7dba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7ff346ae-399e-49d8-87af-3c809cfc7dba None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:24.653476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:24.654406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:24.655238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 238/485] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:24.660839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff346ae-399e-49d8-87af-3c809cfc7dba None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:24.726564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d075030-6de7-41d1-92ea-59818898338e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:24.766415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc90728-9b86-44b1-8b7d-83100cdbf16b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:24.767271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc90728-9b86-44b1-8b7d-83100cdbf16b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:24.767555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc90728-9b86-44b1-8b7d-83100cdbf16b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:24.767826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc90728-9b86-44b1-8b7d-83100cdbf16b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:24.779456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc90728-9b86-44b1-8b7d-83100cdbf16b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-1bc90728-9b86-44b1-8b7d-83100cdbf16b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:24.784662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:24.785580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:24.786442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 248/486] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:24.792480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc90728-9b86-44b1-8b7d-83100cdbf16b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:24.802959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:24.822325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:24.893403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff346ae-399e-49d8-87af-3c809cfc7dba None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:24.932180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77f152b-943d-4830-a227-d5dc832169bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:24.933161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f152b-943d-4830-a227-d5dc832169bd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:24.933981 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f152b-943d-4830-a227-d5dc832169bd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:24.934505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f152b-943d-4830-a227-d5dc832169bd None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:24.935460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77f152b-943d-4830-a227-d5dc832169bd None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:24.935956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77f152b-943d-4830-a227-d5dc832169bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:24.937507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77f152b-943d-4830-a227-d5dc832169bd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:24.949256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 239/487] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:24.951535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84a441c-eac9-4d69-a4c5-c3bb6ca276b8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:24.952216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84a441c-eac9-4d69-a4c5-c3bb6ca276b8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:24.952643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84a441c-eac9-4d69-a4c5-c3bb6ca276b8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:24.954331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc90728-9b86-44b1-8b7d-83100cdbf16b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:24.954513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 240/488] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:24.955433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84a441c-eac9-4d69-a4c5-c3bb6ca276b8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:24.997216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3836c8-1e90-4d8d-ad35-140dcc11efc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:24.998030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3836c8-1e90-4d8d-ad35-140dcc11efc8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:24.998444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3836c8-1e90-4d8d-ad35-140dcc11efc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:24.998958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3836c8-1e90-4d8d-ad35-140dcc11efc8 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:24.999744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3836c8-1e90-4d8d-ad35-140dcc11efc8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:25.000143 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3836c8-1e90-4d8d-ad35-140dcc11efc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:25.001970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3836c8-1e90-4d8d-ad35-140dcc11efc8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:25.011155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 249/489] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:25.012544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef6eb8a-540f-49b7-abc1-9c251e7fdc16 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.012930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef6eb8a-540f-49b7-abc1-9c251e7fdc16 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.013303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef6eb8a-540f-49b7-abc1-9c251e7fdc16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.032918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-aef6eb8a-540f-49b7-abc1-9c251e7fdc16 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:25.034152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-aef6eb8a-540f-49b7-abc1-9c251e7fdc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:25.084051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84a441c-eac9-4d69-a4c5-c3bb6ca276b8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:25.108092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 250/490] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:25.114031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef6eb8a-540f-49b7-abc1-9c251e7fdc16 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.116357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:25.126601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:25.127434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.129288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.129288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.140100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:25.141928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:25.150562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:25.159579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:25.162343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:25.163478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:25.163478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:25.163478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:25.163730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 241/491] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:25.169646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.292845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef6eb8a-540f-49b7-abc1-9c251e7fdc16 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:25.319714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f07f6c-46fa-47d0-b26e-45b04018cd68 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:25.333614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f710ad-1fe8-46b9-8da9-c6bb06c22db5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:25.334295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f710ad-1fe8-46b9-8da9-c6bb06c22db5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.334575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f710ad-1fe8-46b9-8da9-c6bb06c22db5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.334845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f710ad-1fe8-46b9-8da9-c6bb06c22db5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.345710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f710ad-1fe8-46b9-8da9-c6bb06c22db5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:25.347237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f710ad-1fe8-46b9-8da9-c6bb06c22db5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:25.361551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da94845-667e-4d73-aea3-cab88b34502b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-60f710ad-1fe8-46b9-8da9-c6bb06c22db5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:25.362100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:25.362962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:25.363897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:25.363897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:25.363897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:25.363897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 251/492] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:25.364428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83b1b6b-8b5c-4a79-a05e-eef320ad1009 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.364601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83b1b6b-8b5c-4a79-a05e-eef320ad1009 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.364925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83b1b6b-8b5c-4a79-a05e-eef320ad1009 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.365228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da94845-667e-4d73-aea3-cab88b34502b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.365682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da94845-667e-4d73-aea3-cab88b34502b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.366100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da94845-667e-4d73-aea3-cab88b34502b None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.366444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 252/493] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:25.367121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da94845-667e-4d73-aea3-cab88b34502b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:25.367461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06791777-4dd5-459c-af55-06e3210812cf None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.367741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06791777-4dd5-459c-af55-06e3210812cf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.367967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da94845-667e-4d73-aea3-cab88b34502b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:25.368059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06791777-4dd5-459c-af55-06e3210812cf None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.373635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 253/494] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:25.373635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06791777-4dd5-459c-af55-06e3210812cf None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.374128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da94845-667e-4d73-aea3-cab88b34502b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:25.387320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 242/495] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:25.389025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e87575-2635-4c90-9b92-af31a5803b1f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.389309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e87575-2635-4c90-9b92-af31a5803b1f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.389579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e87575-2635-4c90-9b92-af31a5803b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.408018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-89e87575-2635-4c90-9b92-af31a5803b1f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:25.409183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-89e87575-2635-4c90-9b92-af31a5803b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:25.439150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:25.454926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:25.474834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 243/496] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:25.476117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b08bbf-dbbe-47d0-b5b7-ace18f7f6be1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.476403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b08bbf-dbbe-47d0-b5b7-ace18f7f6be1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.476678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b08bbf-dbbe-47d0-b5b7-ace18f7f6be1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.488627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06791777-4dd5-459c-af55-06e3210812cf None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:25.507023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c6b08bbf-dbbe-47d0-b5b7-ace18f7f6be1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:25.508197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c6b08bbf-dbbe-47d0-b5b7-ace18f7f6be1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:25.524267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:25.524878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.525292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.525563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.526226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:25.526526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:25.527881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:25.544886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 254/497] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:25.544886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.584856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 244/498] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:25.586332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b08bbf-dbbe-47d0-b5b7-ace18f7f6be1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.647256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cfed32-ae1f-4711-ad88-7de6d9b37336 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:25.675371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:25.683780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:25.684330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.684615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.684936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.692658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:25.694241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:25.706869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:25.707791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 25 04:53:25.708705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:25.713514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 255/499] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/users/glance => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:25.714393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.720824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:25.725958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b08bbf-dbbe-47d0-b5b7-ace18f7f6be1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:25.765153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c6eec6-21e2-4f10-a376-336c4ed0a73d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:25.765817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c6eec6-21e2-4f10-a376-336c4ed0a73d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.766093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c6eec6-21e2-4f10-a376-336c4ed0a73d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.766363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c6eec6-21e2-4f10-a376-336c4ed0a73d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.778712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c6eec6-21e2-4f10-a376-336c4ed0a73d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-87c6eec6-21e2-4f10-a376-336c4ed0a73d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:25.783388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:25.784298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:25.785125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 245/500] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:25.785833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c6eec6-21e2-4f10-a376-336c4ed0a73d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.830556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7c21d3-c280-4144-9c5c-183c702772d9 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:25.865812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:25.866434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.866710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.866978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.874421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:25.875870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:25.886727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:25.887606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:25.888522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 256/501] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:25.889106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.889181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c6eec6-21e2-4f10-a376-336c4ed0a73d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:25.923705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:25.924300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:25.924577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:25.924853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:25.925577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:25.926001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:25.927346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:25.940222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:25.964181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 246/502] 10.222.0.16 () {56 vars in 1137 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:25.965057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:25.985096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:25.991262 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a830c-734a-44a5-9888-df883ea07465 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.024332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.024985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.025269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.025543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.036014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.037469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:26.052823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:26.053811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:26.054688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:26.054688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:26.054688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:26.054688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 257/503] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:26.054688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.068286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7335ec41-cdc0-4722-a2b0-89f7869f5c39 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.103906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.104505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.104705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.104977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.105661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:26.105967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.107330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.115048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 247/504] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:26.115819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.158518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:26.161399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76665c11-32ce-42e8-9287-06d449568594 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.194190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be596c7-eded-423f-a0ae-a1ceea7f0977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.194814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be596c7-eded-423f-a0ae-a1ceea7f0977 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.195096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be596c7-eded-423f-a0ae-a1ceea7f0977 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.195368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be596c7-eded-423f-a0ae-a1ceea7f0977 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.201814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:26.205804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be596c7-eded-423f-a0ae-a1ceea7f0977 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-1be596c7-eded-423f-a0ae-a1ceea7f0977 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:26.210591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:26.211532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:26.212713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 258/505] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:26.213452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be596c7-eded-423f-a0ae-a1ceea7f0977 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.226102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb715040-cd8f-4b0a-b1c6-0617fa6c7d4c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.261494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.262140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.262421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.262694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.263415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:26.263720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.265136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.282319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 248/506] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:26.283106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.317835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be596c7-eded-423f-a0ae-a1ceea7f0977 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.351642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973cb6e-750a-4e11-b002-6abdd7130cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.352256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973cb6e-750a-4e11-b002-6abdd7130cf7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.352536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973cb6e-750a-4e11-b002-6abdd7130cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.352890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973cb6e-750a-4e11-b002-6abdd7130cf7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.379918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:26.380841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370cc461-eb72-47ff-8a7b-ae178c3ec396 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.413821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.414412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.414689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.414959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.415642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:26.415947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.417350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.419581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:26.419709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9973cb6e-750a-4e11-b002-6abdd7130cf7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.421153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9973cb6e-750a-4e11-b002-6abdd7130cf7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.428733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 249/507] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:26.433080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.517778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 259/508] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:26.518604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973cb6e-750a-4e11-b002-6abdd7130cf7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.529476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd6d94-401b-45a3-bb6e-d78a51c3cf4c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.562397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.563015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.563291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.563561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.574539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.575764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:26.575962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:26.589998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:26.590958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:26.591837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:26.591837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:26.591837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:26.591837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 250/509] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:26.592291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.617234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973cb6e-750a-4e11-b002-6abdd7130cf7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.643987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:26.651736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.652385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.652668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.652937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.653604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:26.656162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.656162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.663097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 260/510] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:26.663843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.694574 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191f9f16-b60c-460d-85d5-e5d5e080b9db None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.728896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc375ea-5f44-427d-b220-28a0c9fb0e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.729497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc375ea-5f44-427d-b220-28a0c9fb0e6d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.729804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc375ea-5f44-427d-b220-28a0c9fb0e6d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.730073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc375ea-5f44-427d-b220-28a0c9fb0e6d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.762435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fafb5ae-f29e-4abc-935b-8e182221560e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:26.784421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:26.796615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4040b55f-e8db-4826-9b5e-cbf9727707ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:26.796757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc375ea-5f44-427d-b220-28a0c9fb0e6d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.797208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4040b55f-e8db-4826-9b5e-cbf9727707ef None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:26.797489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4040b55f-e8db-4826-9b5e-cbf9727707ef None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:26.798629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4040b55f-e8db-4826-9b5e-cbf9727707ef None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:26.798717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc375ea-5f44-427d-b220-28a0c9fb0e6d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.798794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4040b55f-e8db-4826-9b5e-cbf9727707ef None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:26.799921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4040b55f-e8db-4826-9b5e-cbf9727707ef None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:26.844526 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:26.874945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 261/511] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:53:26 2022] POST /identity/v3/users => generated 247 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:26.875903 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4040b55f-e8db-4826-9b5e-cbf9727707ef None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:26.902617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 251/512] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:26.902617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc375ea-5f44-427d-b220-28a0c9fb0e6d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.001773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4040b55f-e8db-4826-9b5e-cbf9727707ef None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.003099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc375ea-5f44-427d-b220-28a0c9fb0e6d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.040676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.041315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.041616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.041925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.044671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.045319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.045646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.045890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.046533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:27.046829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.048237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.053684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.055132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.059977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 252/513] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:27.061048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:27.072466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:27.073376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:27.074270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:27.074270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:27.074270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:27.074270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 262/514] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:27.074270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.075259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:27.108729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:27.169333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6bdbae-8a01-484c-ae0f-dc2dbd285a87 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.187444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030303c3-f828-462b-84c2-f3c22042de80 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.204834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.205654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.206063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.206436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.220376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.220376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.227729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.229033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.229318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.229603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:27.235690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:27.236774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:27.237758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:27.237758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:27.237758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:27.238523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 253/515] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:27.239204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.246446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.247985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:27.263392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:27.264314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:27.265154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:27.265154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:27.265154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:27.265154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 263/516] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:27.265374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.306063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:27.346901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74748129-1ebd-4a28-a927-d16f37a1725c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.353906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:27.373533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06ed4f1-4559-493e-8738-2e9f4dd0802c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.384719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.385356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.385668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.385935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.394500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.395946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:27.408933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:27.409880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:27.410778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 254/517] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/users/cinder => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:27.411340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.412185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.412805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.413082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.413352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.414027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:27.414328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.416259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.432774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 264/518] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:27.432774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.522014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eb328a-6ba2-43a4-b5bd-8c70c404cca2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.535379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b3c5d3-dd91-4200-8366-5fd76b988e8a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.543320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:27.557737 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.558374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.558619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.558880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.559571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:27.559871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.561243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.571433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94283067-7544-4086-a424-890e5ddae946 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.572559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94283067-7544-4086-a424-890e5ddae946 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.572842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94283067-7544-4086-a424-890e5ddae946 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.573063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 255/519] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:27.573948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.577623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94283067-7544-4086-a424-890e5ddae946 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.578322 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94283067-7544-4086-a424-890e5ddae946 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:27.578782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94283067-7544-4086-a424-890e5ddae946 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.580022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94283067-7544-4086-a424-890e5ddae946 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.592813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 265/520] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:27.597889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94283067-7544-4086-a424-890e5ddae946 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.602125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:27.684679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa56f827-aad2-40ca-8ee5-256ab900e78b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.704560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94283067-7544-4086-a424-890e5ddae946 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.719744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.720343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.720620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.720888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.721625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:27.721929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.723349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.739604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.740235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.740512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.740782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.745673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 256/521] 10.222.0.16 () {56 vars in 1137 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:27.745673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.752078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.753667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:27.766708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:27.767712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:27.768608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 266/522] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:27.769102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.788424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:27.835389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:27.856606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0096160-9a53-4849-b648-5361c06a25f7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.879561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2ceefc-9df2-4128-af92-6c6f1adba04f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:27.893563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.894290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.894633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.894977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.902739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.904244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:27.915770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:27.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:27.922509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 257/523] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/users/neutron => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:27.923196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:27.923305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:27.923397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:27.923397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:27.923397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:27.929823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:27.932089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:27.944284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:27.945288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:27.946453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:27.946453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:27.946453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:27.946453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:27.946453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:27.946453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:27.946453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:27.946453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:27.947216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 267/524] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:27.947892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.029773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:28.034254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539f56e3-9626-4e51-b66c-c33175ca2175 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.055202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c8aff-d223-4507-8ce8-5722568b9039 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.070624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b97b85-a2ac-4e5c-a27d-4bec1a07b8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:28.071224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b97b85-a2ac-4e5c-a27d-4bec1a07b8ee None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.071500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b97b85-a2ac-4e5c-a27d-4bec1a07b8ee None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.071866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b97b85-a2ac-4e5c-a27d-4bec1a07b8ee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.079473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:28.092290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:28.092847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.093122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.093391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.094105 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:28.094409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:28.095845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:28.117839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 268/525] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:28.117839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.150695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b97b85-a2ac-4e5c-a27d-4bec1a07b8ee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:28.152147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b97b85-a2ac-4e5c-a27d-4bec1a07b8ee None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:28.231009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8793f756-3005-4ad4-93d2-78cd3fcf2c20 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.260570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:28.270151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 258/526] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:28.270298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:28.270298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.275603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b97b85-a2ac-4e5c-a27d-4bec1a07b8ee None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.275693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.275693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.275693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:28.275693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:28.275693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:28.307749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 269/527] 10.222.0.16 () {56 vars in 1139 bytes} [Thu Aug 25 04:53:28 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:28.308879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.355107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:28.378918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b97b85-a2ac-4e5c-a27d-4bec1a07b8ee None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.413122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bc2f9b-3489-4325-aa19-73bcbaa82573 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.415119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:28.417014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.417293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.417561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.418329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:28.418631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:28.420046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:28.445745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 259/528] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:28.447253 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.452820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:28.453434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.453780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.454036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.454765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=a6cc8f34f9674a94ba23a7157e44a6f2) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:28.455075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:28.456521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:28.462778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 270/529] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:28 2022] GET /identity/v3/role_assignments?user.id=a6cc8f34f9674a94ba23a7157e44a6f2&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:28.463453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.539035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:28.588355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9783335f-05ac-4ee1-a203-24303ad7980a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.612257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca385d7-1b98-4378-89a2-58ef9b92b41c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.615201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:28.627887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:28.628527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.628807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.629081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.640603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:28.642114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:28.652721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd80f1ee-2848-4e4e-a278-94f1cb6d3d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:28.653346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd80f1ee-2848-4e4e-a278-94f1cb6d3d09 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.653657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd80f1ee-2848-4e4e-a278-94f1cb6d3d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.653938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd80f1ee-2848-4e4e-a278-94f1cb6d3d09 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:28.657915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:28.658990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:28.660019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:28.660019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:28.660019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:28.663533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 260/530] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:28.664376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.727064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd80f1ee-2848-4e4e-a278-94f1cb6d3d09 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:28.728729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd80f1ee-2848-4e4e-a278-94f1cb6d3d09 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:28.796453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4498b8-c9b5-4aed-8c46-5571c6d1bac8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.822066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:28.838153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec5685-0da6-49e0-8f78-2580bcf02c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:28.838768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec5685-0da6-49e0-8f78-2580bcf02c30 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.839044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec5685-0da6-49e0-8f78-2580bcf02c30 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.839310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec5685-0da6-49e0-8f78-2580bcf02c30 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.840033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec5685-0da6-49e0-8f78-2580bcf02c30 None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:28.840337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec5685-0da6-49e0-8f78-2580bcf02c30 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:28.841769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec5685-0da6-49e0-8f78-2580bcf02c30 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:28.846963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 271/531] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:28.846963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd80f1ee-2848-4e4e-a278-94f1cb6d3d09 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.847587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 261/532] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:28 2022] GET /identity/v3/role_assignments?user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:28.848663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c256c-ac2a-44b3-ae1d-d64998af2a9b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:28.848947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c256c-ac2a-44b3-ae1d-d64998af2a9b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:28.849214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c256c-ac2a-44b3-ae1d-d64998af2a9b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:28.850448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 262/533] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:28.851151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1c256c-ac2a-44b3-ae1d-d64998af2a9b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:28.855349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:28.984013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1c256c-ac2a-44b3-ae1d-d64998af2a9b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:28.995077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd80f1ee-2848-4e4e-a278-94f1cb6d3d09 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.028370 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.029104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.029387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.029699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.035412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90782f9-57d8-4d21-a3b0-145360447a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.036266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90782f9-57d8-4d21-a3b0-145360447a8a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.036747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90782f9-57d8-4d21-a3b0-145360447a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.037264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90782f9-57d8-4d21-a3b0-145360447a8a None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.038142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90782f9-57d8-4d21-a3b0-145360447a8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:29.038768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90782f9-57d8-4d21-a3b0-145360447a8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:29.040343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90782f9-57d8-4d21-a3b0-145360447a8a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:29.041432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:29.043010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:29.054368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 272/534] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:29.055858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c69b2e-e0b0-4340-ac32-95eca9accedf None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.056855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c69b2e-e0b0-4340-ac32-95eca9accedf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.057330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c69b2e-e0b0-4340-ac32-95eca9accedf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:29.058505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:29.059395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:29.060257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:29.060257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:29.060257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:29.064074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 263/535] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:29.064895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.078852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f8c69b2e-e0b0-4340-ac32-95eca9accedf None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:29.080361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f8c69b2e-e0b0-4340-ac32-95eca9accedf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:29.094954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:29.145839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:29.157203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 273/536] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:29.158891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c69b2e-e0b0-4340-ac32-95eca9accedf None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.175796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2236d6-3560-46b4-86ec-6787de299191 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.223506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.224201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.224429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.224702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.242632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:29.244238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:29.273658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:29.275297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 264/537] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:29.276180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.280068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c69b2e-e0b0-4340-ac32-95eca9accedf None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.355616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.356463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.356891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.357312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.358146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:29.358664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:29.360182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:29.382440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 274/538] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:29.388878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.401235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:29.426765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a012fc5d-8e31-4ef1-a2ce-1f9d9e434881 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.487214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.488066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.488490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.488905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.489807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:29.490274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:29.491853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:29.492349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:29.509492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 265/539] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:29.510684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.548653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11750866-c231-42c1-8b57-62e9a74751f6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.583864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.583864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.584130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.584364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.592122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:29.593430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:29.605537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:29.606504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:29.607371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:29.607371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:29.607371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:29.607371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:29.607371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:29.607371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:29.607371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:29.607371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:29.611457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 275/540] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:29.612363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.620199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327157f8-01a7-4875-af6c-fe69c0a5333a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.657230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d57cc10-021d-4608-a51a-b3cadde38c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.658258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d57cc10-021d-4608-a51a-b3cadde38c0c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.658772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d57cc10-021d-4608-a51a-b3cadde38c0c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.659246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d57cc10-021d-4608-a51a-b3cadde38c0c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.669911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d57cc10-021d-4608-a51a-b3cadde38c0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-5d57cc10-021d-4608-a51a-b3cadde38c0c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:29.675325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:29.676207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:29.677019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:29.679571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 266/541] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:29.680587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d57cc10-021d-4608-a51a-b3cadde38c0c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.690077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:29.724638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92716f4-74bc-483a-be31-fca0e3de0f76 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.740206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:29.762087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.762930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.763793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.764447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.765384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=ec312d29dfa846cbace53e8444b8a527) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:29.765891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:29.767681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:29.773891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 276/542] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/role_assignments?user.id=ec312d29dfa846cbace53e8444b8a527&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:29.775008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.789632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d57cc10-021d-4608-a51a-b3cadde38c0c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.824793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:29.825472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:29.825781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:29.826107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:29.826813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:29.827123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:29.828554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:29.842725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 267/543] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:29.855846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:29.983866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc9093a-81aa-426b-b45a-66a324eb362e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:29.992615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:30.031500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:30.032139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.032426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.032704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.033385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:30.033715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:30.034312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e431cf-f84c-493c-945d-a4b92df53090 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:30.035114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:30.047917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 277/544] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:30.064425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:30.085747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:30.086769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.087640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.088134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.089070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=9220ba3637a542a382ad1d42ff1ff459) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:30.089664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:30.091476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:30.107153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 268/545] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:29 2022] GET /identity/v3/role_assignments?user.id=9220ba3637a542a382ad1d42ff1ff459&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 259 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:30.132875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:30.238075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f8c1cf-581d-476a-bc08-afc6be1ba593 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:30.273938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:30.274567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.275537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.275808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.287219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:30.288667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:30.303640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:30.304608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:30.305538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:30.305538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:30.305538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:30.305538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 278/546] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:30.312189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:30.502976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d512ea00-8267-4cb7-b830-a7a80dc55e94 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:30.539589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:30.546555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abc9ffb-c21f-4824-837c-6d46f394ffa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:30.547159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc9ffb-c21f-4824-837c-6d46f394ffa7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.547434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc9ffb-c21f-4824-837c-6d46f394ffa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.547699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abc9ffb-c21f-4824-837c-6d46f394ffa7 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.548531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abc9ffb-c21f-4824-837c-6d46f394ffa7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:30.548858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abc9ffb-c21f-4824-837c-6d46f394ffa7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:30.550306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abc9ffb-c21f-4824-837c-6d46f394ffa7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:30.567778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 269/547] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:30.573163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:30.617265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cecc05-33ad-4e9a-bc50-6b2c66426c59 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.617265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cecc05-33ad-4e9a-bc50-6b2c66426c59 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.617265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cecc05-33ad-4e9a-bc50-6b2c66426c59 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.617265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 270/548] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:53:30.624330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcfd3d-082f-4337-8d79-6ceb0ca65bb6 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.624813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcfd3d-082f-4337-8d79-6ceb0ca65bb6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.625090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fcfd3d-082f-4337-8d79-6ceb0ca65bb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.655320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-62fcfd3d-082f-4337-8d79-6ceb0ca65bb6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:30.657717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-62fcfd3d-082f-4337-8d79-6ceb0ca65bb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:30.718003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:30.733004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af15ba6-996e-45e1-9792-8f13dc79c6bd None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:30.779943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 271/549] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:30.803300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533e9c21-3a85-4b9a-acbb-4093b7c2b0e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:30.803938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533e9c21-3a85-4b9a-acbb-4093b7c2b0e1 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.804221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533e9c21-3a85-4b9a-acbb-4093b7c2b0e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.804489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533e9c21-3a85-4b9a-acbb-4093b7c2b0e1 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.816376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533e9c21-3a85-4b9a-acbb-4093b7c2b0e1 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:30.817857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533e9c21-3a85-4b9a-acbb-4093b7c2b0e1 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-533e9c21-3a85-4b9a-acbb-4093b7c2b0e1 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:30.838954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:30.839887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:30.840800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 279/550] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:30 2022] GET /identity/v3/users/swift => generated 83 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:30.847516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fcfd3d-082f-4337-8d79-6ceb0ca65bb6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:30.930676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8af7b3c-ddf4-4dbd-acfa-24251026062a None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.931038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8af7b3c-ddf4-4dbd-acfa-24251026062a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.931574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8af7b3c-ddf4-4dbd-acfa-24251026062a None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.936882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 280/551] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:30.939002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ef6ae-6868-4063-a11c-4275838b4838 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:30.939286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ef6ae-6868-4063-a11c-4275838b4838 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:30.939569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ef6ae-6868-4063-a11c-4275838b4838 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:30.957560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c77ef6ae-6868-4063-a11c-4275838b4838 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:30.958885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c77ef6ae-6868-4063-a11c-4275838b4838 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:30.980715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fcfd3d-082f-4337-8d79-6ceb0ca65bb6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.001548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:31.031712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:31.032256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:31.032533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:31.032806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:31.033613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:31.033718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 281/552] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:31.033929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:31.035354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:31.069835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 272/553] 10.222.0.16 () {56 vars in 1135 bytes} [Thu Aug 25 04:53:30 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:31.076053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ef6ae-6868-4063-a11c-4275838b4838 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:31.175677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:31.194701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ef6ae-6868-4063-a11c-4275838b4838 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.194978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:31.246344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b081b1-1fc9-4c3c-82df-31158c9a8c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:31.246972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b081b1-1fc9-4c3c-82df-31158c9a8c27 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:31.247252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b081b1-1fc9-4c3c-82df-31158c9a8c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:31.247526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b081b1-1fc9-4c3c-82df-31158c9a8c27 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:31.325212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bc3354-e384-41d6-8f75-f4bbe99cc268 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.383331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b081b1-1fc9-4c3c-82df-31158c9a8c27 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:31.385133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b081b1-1fc9-4c3c-82df-31158c9a8c27 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:31.389020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494db7c0-5850-418b-b58a-60b677f38574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:31.389682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494db7c0-5850-418b-b58a-60b677f38574 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:31.389962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494db7c0-5850-418b-b58a-60b677f38574 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:31.390231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494db7c0-5850-418b-b58a-60b677f38574 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:31.416913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494db7c0-5850-418b-b58a-60b677f38574 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-494db7c0-5850-418b-b58a-60b677f38574 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:31.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:31.422625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:31.423969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 273/554] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:31.427215 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494db7c0-5850-418b-b58a-60b677f38574 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:31.478523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:31.479182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:31.499031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 282/555] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:31.499926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b081b1-1fc9-4c3c-82df-31158c9a8c27 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:31.535598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494db7c0-5850-418b-b58a-60b677f38574 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.573990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663cd25e-8169-45ae-93c5-67a2c4fd96d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:31.574685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663cd25e-8169-45ae-93c5-67a2c4fd96d0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:31.574973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663cd25e-8169-45ae-93c5-67a2c4fd96d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:31.575752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663cd25e-8169-45ae-93c5-67a2c4fd96d0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:31.589912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663cd25e-8169-45ae-93c5-67a2c4fd96d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-663cd25e-8169-45ae-93c5-67a2c4fd96d0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:31.597247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:31.598169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:31.599484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 274/556] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:31.600155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b081b1-1fc9-4c3c-82df-31158c9a8c27 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.600258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663cd25e-8169-45ae-93c5-67a2c4fd96d0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:31.646667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:31.647281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:31.647562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:31.647837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:31.648934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:31.649245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:31.651650 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:31.665921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 283/557] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:31.666776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:31.716019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:31.738568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663cd25e-8169-45ae-93c5-67a2c4fd96d0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.757954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:31.776962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:31.777489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:31.777879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:31.778299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:31.781457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04866441-43de-4cf8-a14f-40cd4d92d2c4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.790059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:31.791771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:31.812623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:31.813636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:31.814593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:31.814593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:31.814593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:31.814593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 275/558] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:31.815371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:31.826243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:31.827112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:31.827572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:31.828365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:31.829211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:31.830019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:31.832085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:31.844101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 284/559] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:31.845169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:31.928476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f43f0-9176-4033-813e-a028e81e1531 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.973475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:31.974140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:31.974418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:31.974689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:31.980450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93292173-7327-40fe-863a-8e60c08bda20 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:31.988636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:31.991383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:31.993767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:32.014112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:32.015117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:32.015998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:32.015998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:32.015998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:32.022833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 276/560] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:32.022833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.024218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.025503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.028021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.028313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:32.028477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.029605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:32.031306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.034772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.051258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 285/561] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:32.052483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.153666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb991a-cdc2-4955-9f34-c12bf66418d1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.161287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716b3870-88af-4283-b58d-0be994a1346b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.195531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c17909-73e9-4c6d-80be-ea0b2c8ffc12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.196254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c17909-73e9-4c6d-80be-ea0b2c8ffc12 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.197062 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c17909-73e9-4c6d-80be-ea0b2c8ffc12 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.197335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c17909-73e9-4c6d-80be-ea0b2c8ffc12 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.200457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.201068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.201350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.201645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.202289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:32.202594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.203983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.208439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c17909-73e9-4c6d-80be-ea0b2c8ffc12 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.209989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c17909-73e9-4c6d-80be-ea0b2c8ffc12 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.221657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 286/562] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:32.222530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c1c17909-73e9-4c6d-80be-ea0b2c8ffc12 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:32.233908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:32.234863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:32.235769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:32.235769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:32.235769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:32.235769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 277/563] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:32.236733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ba1ca-74be-473d-85a4-e0c38f049773 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.236847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ba1ca-74be-473d-85a4-e0c38f049773 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.237023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ba1ca-74be-473d-85a4-e0c38f049773 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.241718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 278/564] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:32.241718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ba1ca-74be-473d-85a4-e0c38f049773 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.256088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:32.301886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:32.335585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2f94b9-6256-46a6-b77e-32daa0e951de None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.348483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ba1ca-74be-473d-85a4-e0c38f049773 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.370371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.371002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.371281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.371566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.378130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.379575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.383849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172636cc-1b65-4889-850d-2893e82fa04d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.384488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172636cc-1b65-4889-850d-2893e82fa04d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.384762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172636cc-1b65-4889-850d-2893e82fa04d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.385035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172636cc-1b65-4889-850d-2893e82fa04d None admin] PATH_INFO: `/v3/users/glance` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:32.391435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:32.397915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:32.398996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 287/565] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:32.400105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172636cc-1b65-4889-850d-2893e82fa04d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.400194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.400285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172636cc-1b65-4889-850d-2893e82fa04d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-172636cc-1b65-4889-850d-2893e82fa04d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:32.408965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:32.410137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:32.411098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 279/566] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/users/glance => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:32.411723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a85193-9b5a-4437-b9cd-848bbf740bc4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.411723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a85193-9b5a-4437-b9cd-848bbf740bc4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.411723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a85193-9b5a-4437-b9cd-848bbf740bc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.429086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d7a85193-9b5a-4437-b9cd-848bbf740bc4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:32.430299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d7a85193-9b5a-4437-b9cd-848bbf740bc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:32.497072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 280/567] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:32.498578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bb257a-a8b9-45e4-94b3-930526560de0 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.498862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bb257a-a8b9-45e4-94b3-930526560de0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.499135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bb257a-a8b9-45e4-94b3-930526560de0 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.500360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 281/568] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:32.501053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bb257a-a8b9-45e4-94b3-930526560de0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.522479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:32.538409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb5e3b-608e-4847-b981-cfe6bcf55b0a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.553506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:32.577138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.577786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.578070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.578343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.578980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:32.579283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.580657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.597521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 288/569] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:32.597521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.609228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bb257a-a8b9-45e4-94b3-930526560de0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.643133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44e0b87-c002-4a3b-b640-4bcbf8203b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.643745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e0b87-c002-4a3b-b640-4bcbf8203b77 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.644024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e0b87-c002-4a3b-b640-4bcbf8203b77 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.644300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e0b87-c002-4a3b-b640-4bcbf8203b77 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.645012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e0b87-c002-4a3b-b640-4bcbf8203b77 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:32.645322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e0b87-c002-4a3b-b640-4bcbf8203b77 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.646804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e0b87-c002-4a3b-b640-4bcbf8203b77 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.659561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 282/570] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:32.660652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd443915-5e79-4c77-af84-9045075a0441 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.660937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd443915-5e79-4c77-af84-9045075a0441 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.661211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd443915-5e79-4c77-af84-9045075a0441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.678558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bd443915-5e79-4c77-af84-9045075a0441 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:32.679736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bd443915-5e79-4c77-af84-9045075a0441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:32.709487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874cdb3b-1841-4e58-a530-493f71f41ded None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.743977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.744579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.744865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.745119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.745871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:32.746193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.749649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 283/571] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:32.749783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.749891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd443915-5e79-4c77-af84-9045075a0441 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.770071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 289/572] 10.222.0.16 () {56 vars in 1137 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:32.770759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.777900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:32.793150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:32.859022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd443915-5e79-4c77-af84-9045075a0441 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.878197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbb5014-168a-4faa-b1e3-823115211ba0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:32.893906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.894528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.894799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.895065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.902678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.904087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.913734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:32.914388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:32.914765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:32.915172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:32.915686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:32.916821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:32.918044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:32.918694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=c77473684c58457aa53f73a9c30319d0) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:32.918959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:32.920475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:32.921395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 284/573] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:32.922308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:32.934180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 290/574] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/role_assignments?user.id=c77473684c58457aa53f73a9c30319d0&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:32.934180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.009446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:33.047436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870c7be7-c9cb-44af-b952-dc5d566aa37a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.054473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:33.061717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026a8ff1-c462-4256-bd6f-10eabab38780 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.083606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.084380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.084700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.084973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.097211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.099096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.101534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c63761-679d-48c1-a43c-e8e55b778b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.102196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c63761-679d-48c1-a43c-e8e55b778b2f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.102475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c63761-679d-48c1-a43c-e8e55b778b2f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.103289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c63761-679d-48c1-a43c-e8e55b778b2f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:33.113228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:33.114418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:33.115476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:33.115476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:33.115476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:33.115476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 291/575] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:33.115476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.119981 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c63761-679d-48c1-a43c-e8e55b778b2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e2c63761-679d-48c1-a43c-e8e55b778b2f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:33.124787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:33.125930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:33.126959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:33.129738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 285/576] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:33.131805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c63761-679d-48c1-a43c-e8e55b778b2f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.238733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0eb92c-0b94-42ef-804e-1f7e6bf9ab55 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.240295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c63761-679d-48c1-a43c-e8e55b778b2f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.259981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:33.275286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.275995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.276274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.276375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96730b61-e421-42c1-a434-d69ca5ea396f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.276546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.277024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96730b61-e421-42c1-a434-d69ca5ea396f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.277266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:33.277457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96730b61-e421-42c1-a434-d69ca5ea396f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.277551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.277744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96730b61-e421-42c1-a434-d69ca5ea396f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.279240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.288924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96730b61-e421-42c1-a434-d69ca5ea396f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-96730b61-e421-42c1-a434-d69ca5ea396f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:33.294210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:33.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:33.295982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 286/577] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:33.296680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96730b61-e421-42c1-a434-d69ca5ea396f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.300420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 292/578] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:33.302508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.320531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:33.407683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96730b61-e421-42c1-a434-d69ca5ea396f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.412678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6193282-6c1e-4446-b63a-fceb787adb05 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.447311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.447915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.448199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.448470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.448901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.449109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:33.449409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.449504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.449803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.450075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.450730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:33.450825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.451035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.452501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.466790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 287/579] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:33.466790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.467013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 293/580] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:33.467115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.511288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:33.543259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:33.569802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6e3c45-3b9d-45ab-832e-b8b28efc12e0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.571301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a1a60d-6d25-417b-89d3-c90ebd6a6871 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.607175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.607780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.608057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.608158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.608348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.608727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.608972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:33.609066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.609269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.609362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.610650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.618979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 288/581] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:33.623689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.624628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.626120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:33.641995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:33.642945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:33.643850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:33.643850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:33.643850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:33.643850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 294/582] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:33.644636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.738098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29432664-a63b-47af-ae41-67070a1dddc2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.739678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:33.749519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef42d168-b13c-4f3c-8ccb-705dac9a9e03 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.773805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.774362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.774668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.774947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.777904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:33.782321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.782955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.783234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.783506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.790593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.790721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.792106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.792217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:33.804555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:33.806713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 295/583] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:33.807620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:33.809053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:33.809991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:33.811007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:33.811007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:33.811007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:33.813668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 289/584] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:33.814597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.914406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7ce57d-62db-4621-a696-719954521f2c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.931836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee69eb3-e507-4116-978b-d9e268fc2182 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:33.952759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.953461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.953767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.954042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.965847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.966755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:33.967339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.967441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:33.967640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:33.967913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:33.968553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:33.968862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:33.970801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:33.982047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:33.983026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:33.989094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:33.989094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:33.989094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:33.989094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 296/585] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:33.989094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.989413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 290/586] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:33.989413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:33.992607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:34.045412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:34.099033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e5ef02-97b0-493e-bc0f-d78d18a4f63e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.101518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f853ee-c3f3-411c-8e8d-6191f7a04304 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.133493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.134453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.134946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.135401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.136197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.136475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:34.136797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.137076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.137345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.137521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.138006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:34.138306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.139546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.139655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.156974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 297/587] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:34.157140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 291/588] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:34.157219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.157316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.226001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:34.264547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5065e27-5460-45bd-9abc-d7b85b1ab0bc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.266638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37de6cf1-1df9-4e3c-a865-a129275a8bbc None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.274601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:34.299733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.300369 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.300652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.300930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.301200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.301581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:34.301903 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.302005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.302169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.302428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.303301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.309475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.310954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.319933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 292/589] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:34.319933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:34.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:34.327653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:34.328537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:34.328537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:34.328537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:34.328537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:34.328537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:34.328537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:34.328537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:34.328537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:34.332820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 298/590] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:34.334119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.448140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0016bd6e-3e19-46f8-9bb5-92f886ce8aa4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.457775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13758dc4-a999-4e24-bea2-89038329c986 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.474009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:34.486181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac81bfaf-00b7-405a-b4aa-12ab4aaef620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.486842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac81bfaf-00b7-405a-b4aa-12ab4aaef620 None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.487198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac81bfaf-00b7-405a-b4aa-12ab4aaef620 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.487389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac81bfaf-00b7-405a-b4aa-12ab4aaef620 None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/a6cc8f34f9674a94ba23a7157e44a6f2/roles/c9f7a48328df4a7ea7a5981959f395f5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.496136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.496791 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.497066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.497337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.502720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac81bfaf-00b7-405a-b4aa-12ab4aaef620 None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=a6cc8f34f9674a94ba23a7157e44a6f2, role_id=c9f7a48328df4a7ea7a5981959f395f5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.504194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac81bfaf-00b7-405a-b4aa-12ab4aaef620 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.505141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.506600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:34.518580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:34.521035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 293/591] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:34 2022] GET /identity/v3/users/cinder => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:34.522045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.556438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:34.581917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 299/592] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:53:34 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/a6cc8f34f9674a94ba23a7157e44a6f2/roles/c9f7a48328df4a7ea7a5981959f395f5 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:34.582924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac81bfaf-00b7-405a-b4aa-12ab4aaef620 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.652653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f364c8-64fd-406c-bd97-eb9b8613c992 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.704331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.705081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.705453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.705828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.717637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.719538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:34.733915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:34.734929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:34.735897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 294/593] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:34 2022] GET /identity/v3/users/neutron => generated 85 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:34.736484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.743817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac81bfaf-00b7-405a-b4aa-12ab4aaef620 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.794417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.795272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.795597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.796253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.796952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:34.797259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.799281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.815933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 300/594] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:34.816648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.841290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:34.854462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4191bf30-c867-41ca-80cb-04c6f488c2a4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.859804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:34.890706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.891400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.891684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.891960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.892686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:34.892992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.894439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.912046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 295/595] 10.222.0.16 () {56 vars in 1137 bytes} [Thu Aug 25 04:53:34 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:34.917338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:34.929739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56221bb8-7178-4aff-85d3-2269f2b12b68 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:34.966767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d8b2a5-be1b-4e5e-8f60-eb34fee5f74c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:34.967379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8b2a5-be1b-4e5e-8f60-eb34fee5f74c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.967708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8b2a5-be1b-4e5e-8f60-eb34fee5f74c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.967994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8b2a5-be1b-4e5e-8f60-eb34fee5f74c None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.968691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d8b2a5-be1b-4e5e-8f60-eb34fee5f74c None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:34.968996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d8b2a5-be1b-4e5e-8f60-eb34fee5f74c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:34.970453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d8b2a5-be1b-4e5e-8f60-eb34fee5f74c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:34.988518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 301/596] 10.222.0.16 () {56 vars in 1139 bytes} [Thu Aug 25 04:53:34 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:34.989741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d912a93-0a62-4249-9b5e-ab5024719009 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:34.990023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d912a93-0a62-4249-9b5e-ab5024719009 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:34.990292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d912a93-0a62-4249-9b5e-ab5024719009 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:34.991620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 302/597] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:34.996721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d912a93-0a62-4249-9b5e-ab5024719009 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.028319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c771f15-ad09-43ea-bb2f-347b560d7a16 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.064295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1782bbeb-731c-4d24-b597-3e8e31ceb34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.065139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1782bbeb-731c-4d24-b597-3e8e31ceb34c None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.065533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1782bbeb-731c-4d24-b597-3e8e31ceb34c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.065835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1782bbeb-731c-4d24-b597-3e8e31ceb34c None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/eb5ac6ef2f5048d3aed0d7a91abc4cac/roles/c9f7a48328df4a7ea7a5981959f395f5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.083368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1782bbeb-731c-4d24-b597-3e8e31ceb34c None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=eb5ac6ef2f5048d3aed0d7a91abc4cac, role_id=c9f7a48328df4a7ea7a5981959f395f5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:35.084726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1782bbeb-731c-4d24-b597-3e8e31ceb34c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:35.088026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:35.095981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:35.105621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d912a93-0a62-4249-9b5e-ab5024719009 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.138134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.138753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.139030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.139300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.148857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:35.150362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:35.164719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:35.171475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:35.172469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:35.172469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:35.172469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:35.172469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 303/598] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:35.172469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.194948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 296/599] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:53:34 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/eb5ac6ef2f5048d3aed0d7a91abc4cac/roles/c9f7a48328df4a7ea7a5981959f395f5 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:35.203413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc35feb6-2ad8-49d7-b0d0-c6537370f740 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.203934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc35feb6-2ad8-49d7-b0d0-c6537370f740 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.204282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc35feb6-2ad8-49d7-b0d0-c6537370f740 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.262935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cc35feb6-2ad8-49d7-b0d0-c6537370f740 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:35.264265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cc35feb6-2ad8-49d7-b0d0-c6537370f740 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:35.301215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915a0865-3083-4d0a-9ff5-978e82349478 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.340933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 297/600] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:35.342154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc35feb6-2ad8-49d7-b0d0-c6537370f740 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.349741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.350702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.351164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.351614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.370156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:35.372395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:35.376523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:35.392319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:35.393289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:35.394239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:35.394239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:35.394239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:35.400056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 304/601] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:35.402386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.484105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc35feb6-2ad8-49d7-b0d0-c6537370f740 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.527727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f7e6ca-b102-4db4-8283-322f399dc7cc None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.566219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:35.570182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.572500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.572984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.574456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.575313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:35.576445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:35.583340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:35.584708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.585543 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.587926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.588404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.589347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:35.591925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:35.593515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:35.615709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 305/602] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:35.622421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 298/603] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:35.625933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.628117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.701568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:35.728223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df821785-bd31-46ca-965d-55b1211b073f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.762101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5920242a-4b83-4cc7-bcdb-24797535ecbe None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.764462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.765086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.766110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.766110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.772741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:35.774358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:35.785825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:35.786892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:35.787982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:35.787982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:35.787982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:35.787982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:35.787982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:35.787982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:35.787982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:35.787982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:35.794018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 299/604] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:35.796013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.808281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.809402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.809908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.810485 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.819058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:35.820499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:35.836086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:35.837486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:35.838601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:35.838601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:35.838601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:35.838601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:35.838601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:35.838601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:35.838601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:35.838601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:35.840925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 306/605] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:35.841915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.856579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:35.914072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b06aed-64fc-4340-8dc3-a739e7554ce9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.950395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f1e41f-ad3b-4b16-8fc1-2fbaebbdc20a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:35.950974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fb898e-e679-4c3e-a8e4-d4b3159f013e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.951783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb898e-e679-4c3e-a8e4-d4b3159f013e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.952321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb898e-e679-4c3e-a8e4-d4b3159f013e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.952738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb898e-e679-4c3e-a8e4-d4b3159f013e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.964897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fb898e-e679-4c3e-a8e4-d4b3159f013e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:35.968727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d0fb898e-e679-4c3e-a8e4-d4b3159f013e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:35.971491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:35.972614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:35.973696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:35.976067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 300/606] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:35.977099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fb898e-e679-4c3e-a8e4-d4b3159f013e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:35.991515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:35.992252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:35.992713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:35.993087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:35.994034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:35.994459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:35.996161 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:36.011193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 307/607] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:36.012376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:36.086630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fb898e-e679-4c3e-a8e4-d4b3159f013e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:36.094594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:36.121420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ef9b3b-c855-4a09-ad0a-2ba8c89927e9 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:36.123269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:36.123887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:36.124164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:36.124437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:36.125656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:36.125656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:36.126984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:36.140213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 301/608] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:36.141079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:36.162006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:36.162961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:36.163458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:36.163938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:36.164904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:36.165472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:36.167142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:36.176453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 308/609] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:36.177697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:36.222609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:36.259589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980dfe29-bad2-4ad7-9254-a52c9a5150f5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:36.288758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b26a72-3bfd-427a-9800-51c80f9a5b32 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:36.295924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfeb6d8-9fea-49c1-bdb6-7d07c6cce657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:36.296586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfeb6d8-9fea-49c1-bdb6-7d07c6cce657 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:36.296860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfeb6d8-9fea-49c1-bdb6-7d07c6cce657 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:36.297129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfeb6d8-9fea-49c1-bdb6-7d07c6cce657 None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/ec312d29dfa846cbace53e8444b8a527/roles/c9f7a48328df4a7ea7a5981959f395f5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:36.314292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adfeb6d8-9fea-49c1-bdb6-7d07c6cce657 None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=ec312d29dfa846cbace53e8444b8a527, role_id=c9f7a48328df4a7ea7a5981959f395f5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:36.315994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adfeb6d8-9fea-49c1-bdb6-7d07c6cce657 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:36.326989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3957ce5b-b1bc-4166-bed0-06c3f03fc0bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:36.327726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3957ce5b-b1bc-4166-bed0-06c3f03fc0bb None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:36.328012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3957ce5b-b1bc-4166-bed0-06c3f03fc0bb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:36.330093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3957ce5b-b1bc-4166-bed0-06c3f03fc0bb None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/9220ba3637a542a382ad1d42ff1ff459/roles/c9f7a48328df4a7ea7a5981959f395f5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:36.344795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3957ce5b-b1bc-4166-bed0-06c3f03fc0bb None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=9220ba3637a542a382ad1d42ff1ff459, role_id=c9f7a48328df4a7ea7a5981959f395f5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:36.346381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3957ce5b-b1bc-4166-bed0-06c3f03fc0bb None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:36.485121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 302/610] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:53:36 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/ec312d29dfa846cbace53e8444b8a527/roles/c9f7a48328df4a7ea7a5981959f395f5 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:36.485121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfeb6d8-9fea-49c1-bdb6-7d07c6cce657 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:36.516605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 309/611] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:53:36 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/9220ba3637a542a382ad1d42ff1ff459/roles/c9f7a48328df4a7ea7a5981959f395f5 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:36.517837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:36.579803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813b04c-0fe8-43a0-91c8-004d4ed07570 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:36.580146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813b04c-0fe8-43a0-91c8-004d4ed07570 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:36.580349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813b04c-0fe8-43a0-91c8-004d4ed07570 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:36.582084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 310/612] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:53:36.588225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779da2b8-c619-45cd-b0a1-343cfacbe20d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:36.588514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779da2b8-c619-45cd-b0a1-343cfacbe20d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:36.588750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779da2b8-c619-45cd-b0a1-343cfacbe20d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:36.610762 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-779da2b8-c619-45cd-b0a1-343cfacbe20d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:36.611982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-779da2b8-c619-45cd-b0a1-343cfacbe20d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:36.661539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfeb6d8-9fea-49c1-bdb6-7d07c6cce657 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:36.711756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 311/613] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:36.719099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:36.720049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:36.720525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:36.720973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:36.733485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:36.735441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:36.751816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:36.752830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:36.753770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:36.753770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:36.753770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:36.755912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 303/614] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:36.762115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779da2b8-c619-45cd-b0a1-343cfacbe20d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:36.787459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:36.882251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779da2b8-c619-45cd-b0a1-343cfacbe20d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:36.917750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:36.918367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:36.918643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:36.918912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:36.919558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:36.919860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:36.921228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:36.933940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 312/615] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:36.938900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:37.008227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:37.020535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:37.059296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6208d19e-95bb-4f1c-a814-e24c4c6e5c5b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:37.122866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:37.123502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.123832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.124103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] PATH_INFO: `/v3/users/swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.135806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:37.137238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:37.148796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:37.150564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:37.151735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:37.152310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 304/616] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:36 2022] GET /identity/v3/users/swift => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:37.157765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:37.168742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:37.194173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e360d836-4ef0-4039-9d33-ec2c35d3955b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:37.264683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17159469-8ef6-462d-9ce9-0562e8b94581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:37.265246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17159469-8ef6-462d-9ce9-0562e8b94581 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.265529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17159469-8ef6-462d-9ce9-0562e8b94581 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.266057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17159469-8ef6-462d-9ce9-0562e8b94581 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.306262 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17159469-8ef6-462d-9ce9-0562e8b94581 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-17159469-8ef6-462d-9ce9-0562e8b94581 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:37.312409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:37.313349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:37.314237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:37.322146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 313/617] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:37.327112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c9cbfb-2d5a-46c6-915f-49fae1edbafa None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:37.327764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17159469-8ef6-462d-9ce9-0562e8b94581 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:37.361064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:37.371887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb209bb0-74f9-49ab-afcb-a03344b6f4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:37.372984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb209bb0-74f9-49ab-afcb-a03344b6f4c6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.373649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb209bb0-74f9-49ab-afcb-a03344b6f4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.374029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb209bb0-74f9-49ab-afcb-a03344b6f4c6 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.374871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb209bb0-74f9-49ab-afcb-a03344b6f4c6 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:37.375297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb209bb0-74f9-49ab-afcb-a03344b6f4c6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:37.377803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb209bb0-74f9-49ab-afcb-a03344b6f4c6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:37.406576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 305/618] 10.222.0.16 () {56 vars in 1135 bytes} [Thu Aug 25 04:53:37 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:37.408700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9ce8b-80d4-4337-a3d2-389d1d8e6958 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.409617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9ce8b-80d4-4337-a3d2-389d1d8e6958 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.410274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9ce8b-80d4-4337-a3d2-389d1d8e6958 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.412074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 306/619] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:37.413055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac9ce8b-80d4-4337-a3d2-389d1d8e6958 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:37.462495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17159469-8ef6-462d-9ce9-0562e8b94581 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:37.506037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1975c51-fe88-4e10-9476-4c6a3d12d1fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:37.507263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1975c51-fe88-4e10-9476-4c6a3d12d1fb None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.507751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1975c51-fe88-4e10-9476-4c6a3d12d1fb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.508226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1975c51-fe88-4e10-9476-4c6a3d12d1fb None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.509629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1975c51-fe88-4e10-9476-4c6a3d12d1fb None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:37.510527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1975c51-fe88-4e10-9476-4c6a3d12d1fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:37.512420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1975c51-fe88-4e10-9476-4c6a3d12d1fb None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:37.519929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:37.523693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac9ce8b-80d4-4337-a3d2-389d1d8e6958 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:37.525874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 314/620] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:37.528196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f61462-ca08-4d0f-b418-027d27e26ac8 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.528688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f61462-ca08-4d0f-b418-027d27e26ac8 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.532686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f61462-ca08-4d0f-b418-027d27e26ac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.550534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-40f61462-ca08-4d0f-b418-027d27e26ac8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:37.552756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-40f61462-ca08-4d0f-b418-027d27e26ac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:37.568384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:37.569766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.570542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.571033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.585063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:37.587725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:37.611240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:37.612242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:37.613146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:37.613146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:37.613146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:37.615404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 307/621] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:37.616660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:37.645361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 315/622] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:37.646368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f61462-ca08-4d0f-b418-027d27e26ac8 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:37.661987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:37.731455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca882f-a89b-438e-bbf0-4e6e831d78d2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:37.771828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:37.772382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.772660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.772931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.798023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:37.799477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:37.827178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:37.828173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:37.829793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:37.829793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:37.829793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:37.833748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 308/623] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:37.833948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f61462-ca08-4d0f-b418-027d27e26ac8 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:37.834826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:37.872522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:37.873140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:37.873424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:37.873724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:37.874380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:37.874861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:37.875386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:37.877051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:37.890633 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 316/624] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:37.896870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.022377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab331e0c-8507-4e1c-aeef-3da8c60f7b30 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.024610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:38.034910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eadb57a-5062-48b3-9faa-7b184d40ae80 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.068152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.068956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.069561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.069869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.077401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.078008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:38.078129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.078501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.078843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.079468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:38.079580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:38.080060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:38.081612 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:38.093022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:38.094006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:38.094961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:38.094961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:38.094961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:38.094961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:38.094961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:38.094961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:38.094961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:38.094961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:38.095426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 309/625] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:38.095426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.099721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 317/626] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:38.100268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.160392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:38.216902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4348c7f8-8f5c-421b-91e9-7f3a315e83dd None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.220410 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fba18c-9df6-4a8a-8833-0e3fc5a7403e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.260941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a8e727-de39-4bf9-8db9-ccae52947e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.261603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a8e727-de39-4bf9-8db9-ccae52947e42 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.261921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a8e727-de39-4bf9-8db9-ccae52947e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.262308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a8e727-de39-4bf9-8db9-ccae52947e42 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.268335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.269170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.269781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.270221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] PATH_INFO: `/v3/users/glance` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.276268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a8e727-de39-4bf9-8db9-ccae52947e42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:38.280424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d7a8e727-de39-4bf9-8db9-ccae52947e42 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:38.281809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:38.282954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 310/627] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:38.288560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a8e727-de39-4bf9-8db9-ccae52947e42 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.289465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:38.304538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:38.305464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:38.307114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 25 04:53:38.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:38.312294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 318/628] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:38 2022] GET /identity/v3/users/glance => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:38.313309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.427491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a8e727-de39-4bf9-8db9-ccae52947e42 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.436457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c83739-6bbf-401e-affb-f1773c07c7ca None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.448654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:38.469741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.470359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.470985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.471287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.472137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:38.472586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:38.474060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:38.477158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.478628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.479155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.479661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.480539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:38.481061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:38.483921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:38.492935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 311/629] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:38.494028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.501009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 319/630] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:38.502232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.576824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:38.613009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d971362e-59c9-43f9-9889-c6f00fe7e1dd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.622597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deda958-08a7-49bc-88ed-abc1a0260ed3 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.653119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.654327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.654615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.654891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.655635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:38.655941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:38.657371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:38.667263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe8593d-4053-412a-a7ea-977799269df7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.668363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe8593d-4053-412a-a7ea-977799269df7 None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.668962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe8593d-4053-412a-a7ea-977799269df7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.669483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe8593d-4053-412a-a7ea-977799269df7 None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/c77473684c58457aa53f73a9c30319d0/roles/c9f7a48328df4a7ea7a5981959f395f5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.678463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 312/631] 10.222.0.16 () {56 vars in 1137 bytes} [Thu Aug 25 04:53:38 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:38.679698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.691565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe8593d-4053-412a-a7ea-977799269df7 None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=c77473684c58457aa53f73a9c30319d0, role_id=c9f7a48328df4a7ea7a5981959f395f5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:38.693394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe8593d-4053-412a-a7ea-977799269df7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:38.782790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 320/632] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:53:38 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/c77473684c58457aa53f73a9c30319d0/roles/c9f7a48328df4a7ea7a5981959f395f5 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:38.783663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe8593d-4053-412a-a7ea-977799269df7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.785072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e067053e-d7d8-4a53-998f-cfe79e3dc37a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.812979 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:38.833082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:38.842655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3578250f-e5ad-4b78-abb4-f21732c4fbcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.843097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578250f-e5ad-4b78-abb4-f21732c4fbcd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.843428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578250f-e5ad-4b78-abb4-f21732c4fbcd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.843706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578250f-e5ad-4b78-abb4-f21732c4fbcd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.855467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3578250f-e5ad-4b78-abb4-f21732c4fbcd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:38.856940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3578250f-e5ad-4b78-abb4-f21732c4fbcd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-3578250f-e5ad-4b78-abb4-f21732c4fbcd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:38.873758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:38.874802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:38.875710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:38.875710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:38.875710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:38.879939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 313/633] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:38.881380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28ab02c-e59b-40f5-b7f1-2c7d968fe252 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.881760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28ab02c-e59b-40f5-b7f1-2c7d968fe252 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.882033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28ab02c-e59b-40f5-b7f1-2c7d968fe252 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:38.884212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 314/634] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:38.885030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28ab02c-e59b-40f5-b7f1-2c7d968fe252 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:38.926764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe8593d-4053-412a-a7ea-977799269df7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:38.966276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0bb980-af02-4ea9-a18a-4a608264594d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:38.967162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0bb980-af02-4ea9-a18a-4a608264594d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:38.967446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0bb980-af02-4ea9-a18a-4a608264594d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:38.967729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0bb980-af02-4ea9-a18a-4a608264594d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.003613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28ab02c-e59b-40f5-b7f1-2c7d968fe252 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:39.037263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0bb980-af02-4ea9-a18a-4a608264594d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:39.038761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0bb980-af02-4ea9-a18a-4a608264594d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:39.039109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c822139b-2cac-407c-bb2d-22820104332b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:39.039788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c822139b-2cac-407c-bb2d-22820104332b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.040681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c822139b-2cac-407c-bb2d-22820104332b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.041050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c822139b-2cac-407c-bb2d-22820104332b None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.041736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c822139b-2cac-407c-bb2d-22820104332b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:39.042767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c822139b-2cac-407c-bb2d-22820104332b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:39.043386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c822139b-2cac-407c-bb2d-22820104332b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:39.055522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 315/635] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:39.061182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb3f70a-8fc7-4edc-b735-307b893ee26a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.061182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb3f70a-8fc7-4edc-b735-307b893ee26a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.061182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb3f70a-8fc7-4edc-b735-307b893ee26a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.069258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:39.077194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2fb3f70a-8fc7-4edc-b735-307b893ee26a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:39.083617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2fb3f70a-8fc7-4edc-b735-307b893ee26a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:39.114851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:39.150583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 321/636] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:39.150583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc65f38-e534-4068-8bbf-e1c8a3f59a46 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.150583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc65f38-e534-4068-8bbf-e1c8a3f59a46 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.150583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc65f38-e534-4068-8bbf-e1c8a3f59a46 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.150583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 322/637] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:39 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:39.150583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc65f38-e534-4068-8bbf-e1c8a3f59a46 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:39.157335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 316/638] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:39.161633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb3f70a-8fc7-4edc-b735-307b893ee26a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:39.269091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc65f38-e534-4068-8bbf-e1c8a3f59a46 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:39.285334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb3f70a-8fc7-4edc-b735-307b893ee26a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:39.294318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:39.307588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9e469e-7121-45b2-846a-4dcd01605007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:39.308237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e469e-7121-45b2-846a-4dcd01605007 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.308518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e469e-7121-45b2-846a-4dcd01605007 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.308792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e469e-7121-45b2-846a-4dcd01605007 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.317474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9e469e-7121-45b2-846a-4dcd01605007 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:39.319423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9e469e-7121-45b2-846a-4dcd01605007 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:39.329955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:39.331077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.331591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.332054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-3d9e469e-7121-45b2-846a-4dcd01605007 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:39.333460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:39.334512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:39.335544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:39.343671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 323/639] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/users/nova => generated 82 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:39.344811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c89b2f-55b9-4cdc-8580-51555c4d388e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.345103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c89b2f-55b9-4cdc-8580-51555c4d388e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.345378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c89b2f-55b9-4cdc-8580-51555c4d388e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.347311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:39.349067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:39.366333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f3c89b2f-55b9-4cdc-8580-51555c4d388e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:39.367515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f3c89b2f-55b9-4cdc-8580-51555c4d388e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:39.368839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:39.370399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:39.371331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:39.371331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:39.371331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:39.374766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 317/640] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:39.375905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:39.406915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:39.444451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 324/641] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:39.445423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c89b2f-55b9-4cdc-8580-51555c4d388e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:39.500429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51694738-4d8d-47a3-a875-2c6e18630938 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:39.546021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:39.546628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.546908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.547181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.548183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:39.548493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:39.550389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:39.564945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c89b2f-55b9-4cdc-8580-51555c4d388e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:39.576627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 318/642] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:39.576627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:39.588811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:39.611892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:39.612631 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.612918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.613193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.613864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:39.614172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:39.615573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:39.629680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 325/643] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:39.630686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:39.676647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:39.703967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a8d754-0853-4f5e-b015-6c01834131aa None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:39.749321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61634d7-12b1-4ddd-a955-678091484e35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:39.749982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61634d7-12b1-4ddd-a955-678091484e35 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.750256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61634d7-12b1-4ddd-a955-678091484e35 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.750524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61634d7-12b1-4ddd-a955-678091484e35 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.751212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d215f080-b025-49b2-a270-5bc1ef7d9bd3 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:39.761391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61634d7-12b1-4ddd-a955-678091484e35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b61634d7-12b1-4ddd-a955-678091484e35 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:39.766404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:39.767377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:39.768224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 319/644] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:39.768914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61634d7-12b1-4ddd-a955-678091484e35 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:39.788212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5011655-aab4-4682-85e6-50ca1a1b7e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:39.788821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5011655-aab4-4682-85e6-50ca1a1b7e80 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.789098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5011655-aab4-4682-85e6-50ca1a1b7e80 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.789368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5011655-aab4-4682-85e6-50ca1a1b7e80 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.831843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:39.860894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5011655-aab4-4682-85e6-50ca1a1b7e80 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:39.862376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5011655-aab4-4682-85e6-50ca1a1b7e80 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:39.886721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61634d7-12b1-4ddd-a955-678091484e35 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:39.921818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:39.922488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:39.922769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:39.923037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:39.929558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:39.931547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:39.934031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:39.943333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:39.945159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 320/645] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:39.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:39.973353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 326/646] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:39.974404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5011655-aab4-4682-85e6-50ca1a1b7e80 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.047633 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:40.061714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001d580d-3d94-43c8-83e6-f9bd37a63cdc None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.078560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5011655-aab4-4682-85e6-50ca1a1b7e80 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.102415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.103422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.103705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.104112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.104894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:40.105456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.107494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.114046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4b309a-1429-48d1-b825-f087b66b8ef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.114623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4b309a-1429-48d1-b825-f087b66b8ef2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.114905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4b309a-1429-48d1-b825-f087b66b8ef2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.115184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4b309a-1429-48d1-b825-f087b66b8ef2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.118516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 321/647] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:40.119483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.126228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4b309a-1429-48d1-b825-f087b66b8ef2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ea4b309a-1429-48d1-b825-f087b66b8ef2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:40.131560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:40.132654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:40.133516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 327/648] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:40.134828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4b309a-1429-48d1-b825-f087b66b8ef2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.194767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:40.234486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28619b2-c343-4742-9103-2ed5a0a06ca1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.249106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4b309a-1429-48d1-b825-f087b66b8ef2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.271365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.271975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.272256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.272534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.273226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:40.273535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.275021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.280746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:40.286662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.292808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.292954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 322/649] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:40.293068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.293068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.293226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.304139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.305775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:40.319787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:40.320793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:40.322416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:40.322416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:40.322416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:40.322416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 328/650] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:40.322653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.406857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73d1d52-46ac-40f2-8ea7-7ca73f7f2c10 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.429302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6936c04-3fde-47ff-8133-d554a3a0a845 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.432534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:40.444678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.445400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.445773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.446634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.456815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.458400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.464659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.465356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.465733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.466080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.466806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:40.467182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.468610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:40.476184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:40.477362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:40.478517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:40.478517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:40.478517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:40.483509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 323/651] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:40.483509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.483732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 329/652] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:40.483732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.534401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:40.586701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc6540b-33ea-4157-8565-5a3ae69e5b69 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.592003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be22696f-c414-4111-9191-1d5d321b0327 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.622435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.623144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.623416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.623684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.624400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=a6cc8f34f9674a94ba23a7157e44a6f2) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:40.624810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.626257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.628067 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.628746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.629091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.629430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.630196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:40.630565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.632051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.642020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 324/653] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/role_assignments?user.id=a6cc8f34f9674a94ba23a7157e44a6f2&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:40.642020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.659972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 330/654] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:40.661160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.688791 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:40.755654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6450d71-6c0c-4c6e-9385-1cf00d1d89bf None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.780214 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195a2235-93ba-4a6d-a787-3c5433f38662 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.791523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:40.796141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.802672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.802672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.802672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.810780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.812674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.818929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b9ee77-510f-44f4-ba35-8b81b9e0799a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.818929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b9ee77-510f-44f4-ba35-8b81b9e0799a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.819166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b9ee77-510f-44f4-ba35-8b81b9e0799a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.819357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b9ee77-510f-44f4-ba35-8b81b9e0799a None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.820002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b9ee77-510f-44f4-ba35-8b81b9e0799a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:40.820308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b9ee77-510f-44f4-ba35-8b81b9e0799a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.822592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b9ee77-510f-44f4-ba35-8b81b9e0799a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:40.827916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:40.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:40.830187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:40.830187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:40.830187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:40.830187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 325/655] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:40.830187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.837680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 331/656] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:40.839238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b00e39-8923-4fdd-94ea-26a59b12c4c1 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.839734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b00e39-8923-4fdd-94ea-26a59b12c4c1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.840775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b00e39-8923-4fdd-94ea-26a59b12c4c1 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.842828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 332/657] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:40.843715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b00e39-8923-4fdd-94ea-26a59b12c4c1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:40.933224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:40.945067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207372ab-8852-4a44-9b74-1fc539580bb5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.951540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b00e39-8923-4fdd-94ea-26a59b12c4c1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:40.981866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba594921-44a8-4ada-af43-9f330b9529ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.982496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba594921-44a8-4ada-af43-9f330b9529ae None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.982776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba594921-44a8-4ada-af43-9f330b9529ae None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.983045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba594921-44a8-4ada-af43-9f330b9529ae None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.986951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:40.987560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:40.987842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:40.988115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:40.988751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:40.989055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.989974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba594921-44a8-4ada-af43-9f330b9529ae None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:40.990468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:40.991445 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba594921-44a8-4ada-af43-9f330b9529ae None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.006636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 333/658] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:41.006636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ba594921-44a8-4ada-af43-9f330b9529ae None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:41.006865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:41.007713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 326/659] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d46fa1-abcd-46b5-bc08-cd6101fb767f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.008764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d46fa1-abcd-46b5-bc08-cd6101fb767f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.010077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d46fa1-abcd-46b5-bc08-cd6101fb767f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.026677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-57d46fa1-abcd-46b5-bc08-cd6101fb767f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:41.027986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-57d46fa1-abcd-46b5-bc08-cd6101fb767f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:41.038580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:41.101646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 327/660] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:41.101646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d46fa1-abcd-46b5-bc08-cd6101fb767f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.122528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da97aab-e63f-4152-8ff0-80db2bb1065d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.162524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.163370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.163749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.164119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.172148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:41.173889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:41.190549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:41.191539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:41.192506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 334/661] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/users/cinder => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:41.193121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.215937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:41.249448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d46fa1-abcd-46b5-bc08-cd6101fb767f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.287046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.287796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.288049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.288481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:41.289222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.297049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:41.298653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.303739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81914c8e-2d00-4a2b-8af5-7e7835875670 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:41.310857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:41.312088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 25 04:53:41.313291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:41.315308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 328/662] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/users/neutron => generated 85 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:41.316211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.348513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.349835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.350530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.350948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.351945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:41.352797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:41.354831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.369920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 335/663] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:41.371333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.441605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e8d874-734e-44cf-861e-ad522d171c74 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.478148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.478955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.479228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.479491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.480325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:41.480503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadc1613-2926-4c68-baa4-b897e5cbd395 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.480945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:41.482078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.491924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:41.501629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 329/664] 10.222.0.16 () {56 vars in 1137 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:41.502104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.519583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.520205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.520464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.520731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.521443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:41.521776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:41.523242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.545518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 336/665] 10.222.0.16 () {56 vars in 1139 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:41.545518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.548957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:41.608771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbb8018-d909-45ac-b63f-375c24301bf0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.644062 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.644529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf463fa-665a-4077-bec0-6daaf33c658b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.644662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.644934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.645205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.645963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:41.646275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:41.647685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.661627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 330/666] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/role_assignments?user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:41.661627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.699142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f80616-c3cd-4e1a-b529-5db42045d6f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.700145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f80616-c3cd-4e1a-b529-5db42045d6f5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.700492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f80616-c3cd-4e1a-b529-5db42045d6f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.700857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f80616-c3cd-4e1a-b529-5db42045d6f5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.717608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f80616-c3cd-4e1a-b529-5db42045d6f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-28f80616-c3cd-4e1a-b529-5db42045d6f5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:41.725136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:41.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:41.727381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:41.729264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 337/667] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:41.730100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f80616-c3cd-4e1a-b529-5db42045d6f5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:41.749995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:41.793629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39acfcc7-7188-483a-b8f5-864ac7865dac None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.804630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:41.834113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b85ad-e819-48f6-8ee0-39e283d5b81a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.834822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b85ad-e819-48f6-8ee0-39e283d5b81a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.835164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b85ad-e819-48f6-8ee0-39e283d5b81a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.835497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b85ad-e819-48f6-8ee0-39e283d5b81a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.864173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f80616-c3cd-4e1a-b529-5db42045d6f5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:41.918326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68440fdd-44f0-40e3-893e-20c446fc5fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:41.918957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68440fdd-44f0-40e3-893e-20c446fc5fdd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:41.919236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68440fdd-44f0-40e3-893e-20c446fc5fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:41.919508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68440fdd-44f0-40e3-893e-20c446fc5fdd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:41.921214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795b85ad-e819-48f6-8ee0-39e283d5b81a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:41.922704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795b85ad-e819-48f6-8ee0-39e283d5b81a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.994148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68440fdd-44f0-40e3-893e-20c446fc5fdd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:41.995718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68440fdd-44f0-40e3-893e-20c446fc5fdd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:41.998149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.038076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 331/668] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:42.038996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b85ad-e819-48f6-8ee0-39e283d5b81a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.078076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.098915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 338/669] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:42.106132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68440fdd-44f0-40e3-893e-20c446fc5fdd None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.145421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b85ad-e819-48f6-8ee0-39e283d5b81a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.182713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.183329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.183605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.183883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.184648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:42.184956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.186472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.200324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 332/670] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:42.201304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.204616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68440fdd-44f0-40e3-893e-20c446fc5fdd None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.250807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.253638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.254413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.254792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.255167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.266338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.268052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:42.283901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:42.285091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:42.286242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:42.286242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:42.286242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:42.287622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 339/671] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:42.289087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.309705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.314670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf87afe-6bef-4288-aa22-9314d14be802 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.351018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.351681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.351961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.352289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.363354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.364869 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:42.380447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:42.381692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:42.382732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:42.382732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:42.382732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:42.382732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 333/672] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:42.382970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.400966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0e65c9-5d06-4422-adda-3f80149d74cf None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.436475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.437111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.437392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.437695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.448404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.449901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:42.465354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:42.466355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:42.467257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:42.467257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:42.467257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:42.467257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 340/673] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:42.467783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.490169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.503077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32503e8f-70c0-49af-a7aa-7f61ccdc34ef None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.545405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.547951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.548813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.549365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.549836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.554292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:42.554292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.554292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.564597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 334/674] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:42.565783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.592607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36af032b-0349-4615-8057-51ffc610d9e8 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.636541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.637155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.637435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.637755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.639054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:42.639401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.641080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.656897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 341/675] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:42.658045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.674967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c39375-c48d-4572-b5ce-ca922eae565b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.715709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.716315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.716587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.716862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.717498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:42.717864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.719230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.722232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.736776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 335/676] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:42.737835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.770841 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22291623-9b44-46ca-b4c3-f39d166f066a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.798969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.806505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b608a5-3e37-4116-ae60-e2b56f143f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.807129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b608a5-3e37-4116-ae60-e2b56f143f6e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.807404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b608a5-3e37-4116-ae60-e2b56f143f6e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.811820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b608a5-3e37-4116-ae60-e2b56f143f6e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.814337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b608a5-3e37-4116-ae60-e2b56f143f6e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.815792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b608a5-3e37-4116-ae60-e2b56f143f6e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-87b608a5-3e37-4116-ae60-e2b56f143f6e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:42.828455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:42.829498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:42.830437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 342/677] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:42.830946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc3de8-797d-4999-84cd-ad92ff9a4ffd None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.830946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc3de8-797d-4999-84cd-ad92ff9a4ffd None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.831230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc3de8-797d-4999-84cd-ad92ff9a4ffd None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.836420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 343/678] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:42.836420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbc3de8-797d-4999-84cd-ad92ff9a4ffd None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.865315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed024981-41c6-47b2-bc0b-0589fce1b668 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.901780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.902351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.902629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.902906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:42.909529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:42.911001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:42.922380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:42.923293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 336/679] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:42.924204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:42.949603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbc3de8-797d-4999-84cd-ad92ff9a4ffd None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:42.983594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:42.988004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f0e06a-f2c9-44e2-a8fc-c0826d8c37fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:42.988653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f0e06a-f2c9-44e2-a8fc-c0826d8c37fd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:42.989049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f0e06a-f2c9-44e2-a8fc-c0826d8c37fd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:42.989346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f0e06a-f2c9-44e2-a8fc-c0826d8c37fd None admin] PATH_INFO: `/v3/users/swift` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.002577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f0e06a-f2c9-44e2-a8fc-c0826d8c37fd None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.004029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f0e06a-f2c9-44e2-a8fc-c0826d8c37fd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-a1f0e06a-f2c9-44e2-a8fc-c0826d8c37fd None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:43.018001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:43.018995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:43.019935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:43.024671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 344/680] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/users/swift => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:43.025746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f4320b-9eab-4a87-820d-fb42793a3bd4 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.026033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f4320b-9eab-4a87-820d-fb42793a3bd4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.026307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f4320b-9eab-4a87-820d-fb42793a3bd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.050193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ed5646-4fe5-45a6-bd8a-ad248c84539a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:43.052525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-24f4320b-9eab-4a87-820d-fb42793a3bd4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:43.053924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-24f4320b-9eab-4a87-820d-fb42793a3bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:43.093643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:43.095041 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.095630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.096096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.097133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:43.097935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.100038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.116054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:43.118298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 337/681] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:43.119350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:43.149969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 345/682] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:43.151127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f4320b-9eab-4a87-820d-fb42793a3bd4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:43.265895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497017f6-15c0-4797-815e-b9e42f7eb3b1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:43.301725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:43.307642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f4320b-9eab-4a87-820d-fb42793a3bd4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:43.347487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:43.348472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.349972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.351184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.352618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:43.354594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.357165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.366271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:43.367229 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.370822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.371346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.372299 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:43.372831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.374552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.394151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 338/683] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:43.395342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:43.417513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 346/684] 10.222.0.16 () {56 vars in 1135 bytes} [Thu Aug 25 04:53:43 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:43.418743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:43.460439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:43.524441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cf16bb-5be2-4597-ac1e-fd2dcc48c6fb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:43.527799 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf451d6-6fc1-4d77-997b-78a7ce17cbfe None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:43.558346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:43.558866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.559143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.559415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.560135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=ec312d29dfa846cbace53e8444b8a527) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:43.560445 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.561893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.563941 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:43.564663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.564940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.565223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.565947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=9220ba3637a542a382ad1d42ff1ff459) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:43.566254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.567657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.577915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 339/685] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:43 2022] GET /identity/v3/role_assignments?user.id=ec312d29dfa846cbace53e8444b8a527&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:43.577915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:43.584530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 347/686] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:43 2022] GET /identity/v3/role_assignments?user.id=9220ba3637a542a382ad1d42ff1ff459&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:43.585532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:43.626583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:43.708005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a4ae77-8c62-46c3-8d06-5a1d3c689436 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:43.735080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:43.738017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b93f54a-e18f-4d2e-91af-fc2dc5408df8 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:43.753121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76775e26-f641-484e-8a22-dcb748375fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:43.754168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76775e26-f641-484e-8a22-dcb748375fd3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.754649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76775e26-f641-484e-8a22-dcb748375fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.755103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76775e26-f641-484e-8a22-dcb748375fd3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.777912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c668c6-d84e-429e-9daf-430d3bc12af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:43.778564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c668c6-d84e-429e-9daf-430d3bc12af8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.778842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c668c6-d84e-429e-9daf-430d3bc12af8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.779113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c668c6-d84e-429e-9daf-430d3bc12af8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.790146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c668c6-d84e-429e-9daf-430d3bc12af8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.791623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c668c6-d84e-429e-9daf-430d3bc12af8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-31c668c6-d84e-429e-9daf-430d3bc12af8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:43.809576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:43.810644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:43.811567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:43.811567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:43.811567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:43.811972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 348/687] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:43.813132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75897abf-bf8b-4f63-a21a-673a2aa5ed53 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.813325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75897abf-bf8b-4f63-a21a-673a2aa5ed53 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.813622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75897abf-bf8b-4f63-a21a-673a2aa5ed53 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.819786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 349/688] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:43 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:43.820576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75897abf-bf8b-4f63-a21a-673a2aa5ed53 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:43.847271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76775e26-f641-484e-8a22-dcb748375fd3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.848962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76775e26-f641-484e-8a22-dcb748375fd3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.899907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:43.933339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75897abf-bf8b-4f63-a21a-673a2aa5ed53 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:43.968423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:43.969030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.969306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.969576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.970238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:43.970537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:43.971906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:43.972843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 340/689] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:43.974564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54cee4b-3fd4-4613-a538-a9798e2facaf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:43.975076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54cee4b-3fd4-4613-a538-a9798e2facaf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:43.975522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54cee4b-3fd4-4613-a538-a9798e2facaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:43.984028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 350/690] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:43.984863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:43.997236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e54cee4b-3fd4-4613-a538-a9798e2facaf None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:43.999322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e54cee4b-3fd4-4613-a538-a9798e2facaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:44.005823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:44.078988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 341/691] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:44.080191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54cee4b-3fd4-4613-a538-a9798e2facaf None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:44.105198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e8f655-38fa-4406-9921-1e46f706fe79 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.154617 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.155165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.155445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.155716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.166763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:44.168269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:44.172572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:44.183562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:44.192584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:44.193573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:44.193573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:44.193573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:44.193573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 351/692] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:44.193573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:44.208774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54cee4b-3fd4-4613-a538-a9798e2facaf None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.246882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05840f-2745-48c4-b8a0-aeda5dff5b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.247763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05840f-2745-48c4-b8a0-aeda5dff5b9a None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.248192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05840f-2745-48c4-b8a0-aeda5dff5b9a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.248649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05840f-2745-48c4-b8a0-aeda5dff5b9a None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.249440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05840f-2745-48c4-b8a0-aeda5dff5b9a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:44.251302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da05840f-2745-48c4-b8a0-aeda5dff5b9a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:44.274063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:44.313113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77edba73-8898-4e0e-9a89-41b7d2c18607 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.344260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 342/693] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:53:44 2022] POST /identity/v3/users => generated 254 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:44.359846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.360725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.365049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.365049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.365049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:44.365049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:44.365049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:44.379440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 352/694] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:44.384667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05840f-2745-48c4-b8a0-aeda5dff5b9a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:44.471224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:44.502654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05840f-2745-48c4-b8a0-aeda5dff5b9a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.532677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:44.543239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.546454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.546454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.546454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.551861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:44.553466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:44.568219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:44.569158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:44.570087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 343/695] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:44.576004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:44.581820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11094859-7a5c-4fb4-9ac1-9371235185e9 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.608264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:44.620492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a212ed-fdd4-4a2b-9f8f-b73354ba8aaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.621436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a212ed-fdd4-4a2b-9f8f-b73354ba8aaa None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.621944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a212ed-fdd4-4a2b-9f8f-b73354ba8aaa None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.622456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a212ed-fdd4-4a2b-9f8f-b73354ba8aaa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.634267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a212ed-fdd4-4a2b-9f8f-b73354ba8aaa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-25a212ed-fdd4-4a2b-9f8f-b73354ba8aaa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:44.639215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:44.640133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:44.640975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:44.643597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 353/696] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:44.650566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a212ed-fdd4-4a2b-9f8f-b73354ba8aaa None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:44.691105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e51bd6-ecdf-4a75-9c6e-5594fb2936e6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.729234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.729869 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.730146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.730418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.731244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:44.731553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:44.733344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:44.747957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 344/697] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:44.754508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:44.755872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a212ed-fdd4-4a2b-9f8f-b73354ba8aaa None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.793287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.794313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.794593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.794869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.795539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:44.795845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:44.797219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:44.807055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:44.816477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 354/698] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:44.822618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:44.868831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86553e3-649d-4e6e-a0a4-26d31ddefe3a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.875700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:44.907292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.908681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.909200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.909791 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.910835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=c77473684c58457aa53f73a9c30319d0) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:44.911508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:44.913389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:44.926059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 345/699] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:44 2022] GET /identity/v3/role_assignments?user.id=c77473684c58457aa53f73a9c30319d0&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:44.939235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de31d927-6a25-41d9-9d3d-b967d4662c81 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:44.978181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:44.978839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:44.979116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:44.979386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:44.992146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:44.993792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:45.010309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:45.011320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:45.012219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:45.012219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:45.012219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:45.012219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 355/700] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:45.017507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:45.246761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a798a37-e263-471e-84b2-28ce66ae4f94 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:45.258124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:45.283889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:45.284745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:45.285203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:45.285640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:45.286427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:45.286867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:45.288437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:45.301640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 346/701] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:45.307786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:45.410019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:45.413948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fc3b8c-0427-4e02-9649-9ff9bcec85e3 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:45.450643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:45.451306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:45.451581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:45.451896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:45.459289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:45.460733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:45.473178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:45.474168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:45.475192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 356/702] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:53:45 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:45.480812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:45.590230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ac6274-6826-4938-9620-674f95a51112 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:45.626339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46efc328-74b0-4d59-867b-776878fe6965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:45.626949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46efc328-74b0-4d59-867b-776878fe6965 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:45.627222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46efc328-74b0-4d59-867b-776878fe6965 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:45.627488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46efc328-74b0-4d59-867b-776878fe6965 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:45.628212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46efc328-74b0-4d59-867b-776878fe6965 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:45.628511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46efc328-74b0-4d59-867b-776878fe6965 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:45.630007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46efc328-74b0-4d59-867b-776878fe6965 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:45.654727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 347/703] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:53:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:45.660390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:45.819447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54e6bf-1444-42fe-bd93-fbc09a925b8a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:45.819745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54e6bf-1444-42fe-bd93-fbc09a925b8a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:45.819934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac54e6bf-1444-42fe-bd93-fbc09a925b8a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:45.822508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 348/704] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:45.826944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79caa0b9-6c3f-4d2b-aef9-5aec01eacc2a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:45.827224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79caa0b9-6c3f-4d2b-aef9-5aec01eacc2a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:45.827491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79caa0b9-6c3f-4d2b-aef9-5aec01eacc2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:45.848418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-79caa0b9-6c3f-4d2b-aef9-5aec01eacc2a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:45.849809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-79caa0b9-6c3f-4d2b-aef9-5aec01eacc2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:45.881992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:45.904523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4f33d1-6a63-4530-be13-fbb0554eeb28 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:45.933731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 349/705] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 25 04:53:45.935079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b11662-be71-423c-90c2-9388fd40f904 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:45.935367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b11662-be71-423c-90c2-9388fd40f904 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:45.935640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b11662-be71-423c-90c2-9388fd40f904 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:45.937174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 350/706] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:45.943412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70964cf-a452-4929-8d2f-8ea7d7596ccc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:45.943695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70964cf-a452-4929-8d2f-8ea7d7596ccc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:45.943970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70964cf-a452-4929-8d2f-8ea7d7596ccc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:45.970828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e02c4ce-60bf-4c6a-9d2a-e776a3d72f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:45.971451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e02c4ce-60bf-4c6a-9d2a-e776a3d72f6c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:45.971728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e02c4ce-60bf-4c6a-9d2a-e776a3d72f6c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:45.972000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e02c4ce-60bf-4c6a-9d2a-e776a3d72f6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:45.982910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b70964cf-a452-4929-8d2f-8ea7d7596ccc None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:45.984152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b70964cf-a452-4929-8d2f-8ea7d7596ccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:46.025948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:46.098901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e02c4ce-60bf-4c6a-9d2a-e776a3d72f6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:46.100274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e02c4ce-60bf-4c6a-9d2a-e776a3d72f6c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:46.107481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 351/707] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:46.208791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 357/708] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 549 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:46.215031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70964cf-a452-4929-8d2f-8ea7d7596ccc None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:46.223664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e02c4ce-60bf-4c6a-9d2a-e776a3d72f6c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:46.300322 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:46.325417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:46.341546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70964cf-a452-4929-8d2f-8ea7d7596ccc None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:46.347941 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e02c4ce-60bf-4c6a-9d2a-e776a3d72f6c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:46.384811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:46.385877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:46.386381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:46.386849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] PATH_INFO: `/v3/services/network` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:46.387626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:46.389188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:46.392551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:46.393558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:46.393862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:46.394130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:46.405776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:46.406826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:46.407673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:46.420026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 358/709] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:46 2022] GET /identity/v3/services/network => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:46.422915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:46.424367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:46.426474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:46.453945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:46.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:46.455875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:46.455875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:46.455875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:46.461690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 352/710] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:46.465341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:46.594241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e3e52d-f11d-4ec0-94ac-828c04698975 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:46.616894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107b011a-f029-4b48-a9bf-084d87383782 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:46.623632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:46.638364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:46.638921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:46.639184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:46.639451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:46.640089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] RBAC: Adding query filter params (name=network) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:46.641009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:46.642714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:46.653618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 359/711] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:53:46 2022] GET /identity/v3/services?name=network => generated 124 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:46.653618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:46.661732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:46.662694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:46.663117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:46.663531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:46.664513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:46.665019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:46.666763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:46.677338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:46.682699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 353/712] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:46.683736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:46.782592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f442b-05fd-4f4a-8aba-43b132db2123 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:46.791359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c9c51b-579a-45b5-8ba4-7a1c30bae110 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:46.819760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:46.820620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:46.821146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:46.821513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:46.822221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:46.824010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:46.826154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:46.826780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:46.827054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:46.827322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:46.827959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] RBAC: Adding query filter params (type=network) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:46.828256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:46.829780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:46.834282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:46.835408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:46.838589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 360/713] 10.222.0.16 () {56 vars in 1129 bytes} [Thu Aug 25 04:53:46 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:46.840449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:46.845993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 354/714] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:53:46 2022] GET /identity/v3/services?type=network => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:46.847439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:46.895813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:46.951066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:46.960346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ebc529-e220-413d-8351-b323300d46e5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:47.006568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:47.009110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.009731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.010779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.013555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e67d6-46ad-4071-ae9c-abb8157df458 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:47.027706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:47.029225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:47.044141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:47.045338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:47.046599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 361/715] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:47.050081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:47.073877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:47.077060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.077787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.078288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.079277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:47.079920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:47.086218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:47.091999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 355/716] 10.222.0.16 () {56 vars in 1124 bytes} [Thu Aug 25 04:53:46 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:47.098072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:47.160394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b1e07b-41f3-4a44-9d9a-ee17eac813b2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:47.173421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:47.198848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b20cd1d-8cd3-4cd1-8f7f-8b7d735b629e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:47.199481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20cd1d-8cd3-4cd1-8f7f-8b7d735b629e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.199817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20cd1d-8cd3-4cd1-8f7f-8b7d735b629e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.200091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b20cd1d-8cd3-4cd1-8f7f-8b7d735b629e None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.200783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b20cd1d-8cd3-4cd1-8f7f-8b7d735b629e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:47.201087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b20cd1d-8cd3-4cd1-8f7f-8b7d735b629e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:47.202563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b20cd1d-8cd3-4cd1-8f7f-8b7d735b629e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:47.219520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 362/717] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:47.221094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff87a8d-48c9-4696-9986-f634883bf0ee None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.221094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff87a8d-48c9-4696-9986-f634883bf0ee None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.221094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff87a8d-48c9-4696-9986-f634883bf0ee None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.222343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 363/718] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:47.224115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff87a8d-48c9-4696-9986-f634883bf0ee None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:47.256784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e296e23-985b-4a6f-898c-bd476cc856cf None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:47.293906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:47.304086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b546462-268c-4d4c-8821-3ad098f103de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:47.305201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b546462-268c-4d4c-8821-3ad098f103de None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.306200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b546462-268c-4d4c-8821-3ad098f103de None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.306200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b546462-268c-4d4c-8821-3ad098f103de None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.306812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b546462-268c-4d4c-8821-3ad098f103de None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:47.307365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b546462-268c-4d4c-8821-3ad098f103de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:47.310959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b546462-268c-4d4c-8821-3ad098f103de None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:47.318145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 356/719] 10.222.0.16 () {56 vars in 1124 bytes} [Thu Aug 25 04:53:47 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:47.321244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee830c8-8dab-4afc-bdd6-57df7edab696 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.321815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee830c8-8dab-4afc-bdd6-57df7edab696 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.322769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee830c8-8dab-4afc-bdd6-57df7edab696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.345733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff87a8d-48c9-4696-9986-f634883bf0ee None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:47.349995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8ee830c8-8dab-4afc-bdd6-57df7edab696 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:47.351427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8ee830c8-8dab-4afc-bdd6-57df7edab696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:47.387462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:47.388174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.388450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.388718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.389655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:47.389944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:47.391617 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:47.399691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 364/720] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:47 2022] GET /identity/v3/role_assignments?user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:47.441304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 357/721] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:47.761579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:47.806944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:47.895129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87cbc9a-da88-4ac9-ba2b-9956505228e7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:47.940450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1776a50f-2593-44fe-8cef-657470ba3116 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.940767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1776a50f-2593-44fe-8cef-657470ba3116 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.940998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1776a50f-2593-44fe-8cef-657470ba3116 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.942735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 358/722] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:47.946611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0dcab-5f68-4622-97b5-17811efb2d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:47.947358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0dcab-5f68-4622-97b5-17811efb2d25 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.947606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0dcab-5f68-4622-97b5-17811efb2d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.947868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea0dcab-5f68-4622-97b5-17811efb2d25 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.948397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ed4c50-d0c9-4200-912b-e9ac230fb2dd None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:47.948678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ed4c50-d0c9-4200-912b-e9ac230fb2dd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:47.948945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ed4c50-d0c9-4200-912b-e9ac230fb2dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:47.965815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea0dcab-5f68-4622-97b5-17811efb2d25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:47.967857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c5ed4c50-d0c9-4200-912b-e9ac230fb2dd None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:47.969038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c5ed4c50-d0c9-4200-912b-e9ac230fb2dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-dea0dcab-5f68-4622-97b5-17811efb2d25 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:47.970497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:47.971457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:47.972673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 365/723] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:47.983811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0dcab-5f68-4622-97b5-17811efb2d25 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:48.079689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 359/724] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:48.119699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:48.126346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea0dcab-5f68-4622-97b5-17811efb2d25 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:48.160450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:48.161062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:48.161339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:48.161708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:48.162285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:48.162580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:48.163958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:48.174442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 366/725] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:48.183005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ed4c50-d0c9-4200-912b-e9ac230fb2dd None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:48.251823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:48.297836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ed4c50-d0c9-4200-912b-e9ac230fb2dd None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:48.338897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:48.339455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:48.339792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:48.339990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:48.350384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:48.351823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:48.365708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:48.366654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:48.368296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:48.368296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:48.368296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:48.368296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 360/726] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:48.374981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:48.478105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:48.478447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f5cf82-5028-48cf-8a72-aa0dc709fcde None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:48.549619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:48.550249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:48.550527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:48.550794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:48.551432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:48.551729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:48.553089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:48.579122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 367/727] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:48.584470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:48.614123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6402f143-fa44-42f2-839b-39aa47577e45 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:48.627510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:48.637987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:48.669409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57398ce0-04e1-47dc-970c-293080600777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:48.670238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57398ce0-04e1-47dc-970c-293080600777 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:48.670514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57398ce0-04e1-47dc-970c-293080600777 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:48.670780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57398ce0-04e1-47dc-970c-293080600777 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:48.687565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57398ce0-04e1-47dc-970c-293080600777 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-57398ce0-04e1-47dc-970c-293080600777 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:48.693023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:48.693996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:48.694851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 361/728] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:48.700553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57398ce0-04e1-47dc-970c-293080600777 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:48.813002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d8fac-4de6-4898-b954-245b5149c8ef None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:48.853091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57398ce0-04e1-47dc-970c-293080600777 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:48.897152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:48.898277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:48.898557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:48.899355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:48.903272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:48.903872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:48.904146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:48.904413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:48.905069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:48.905363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:48.906739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:48.915383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:48.916818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:48.926907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 362/729] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:48.931354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:48.940169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:48.941102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:53:48.942073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:48.947158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 368/730] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:53:48 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:48.951618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:48.979830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:49.088897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:49.095547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a122ceaf-5888-47f9-8625-4e5bf5638fd0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:49.142666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae5b2ac-db59-4bec-ac24-7e6a9aefff2c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:49.145131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:49.145797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.146077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.146408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.164569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:49.166695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:49.184630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:49.185694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:49.186586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:49.186586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:49.186586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:49.186586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 363/731] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:49.191844 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:49.209617 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eab4b69-3cfe-4514-a1a8-ef75debfaa6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:49.210595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab4b69-3cfe-4514-a1a8-ef75debfaa6e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.211087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab4b69-3cfe-4514-a1a8-ef75debfaa6e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.211605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab4b69-3cfe-4514-a1a8-ef75debfaa6e None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.212946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eab4b69-3cfe-4514-a1a8-ef75debfaa6e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:49.213452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eab4b69-3cfe-4514-a1a8-ef75debfaa6e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:49.215101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eab4b69-3cfe-4514-a1a8-ef75debfaa6e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:49.245360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 369/732] 10.222.0.16 () {56 vars in 1149 bytes} [Thu Aug 25 04:53:48 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:49.247010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d36222-efb4-4845-b6e6-6b9a2fefa396 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.247494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d36222-efb4-4845-b6e6-6b9a2fefa396 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.247972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d36222-efb4-4845-b6e6-6b9a2fefa396 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.249618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 370/733] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:49.250512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d36222-efb4-4845-b6e6-6b9a2fefa396 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:49.302632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:49.327123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1df043-b7e8-4684-a51e-a73fa0801ce8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:49.370460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f44736-7e54-4209-9ba8-efcbf094dd51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:49.371918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f44736-7e54-4209-9ba8-efcbf094dd51 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.372396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f44736-7e54-4209-9ba8-efcbf094dd51 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.372517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d36222-efb4-4845-b6e6-6b9a2fefa396 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:49.372977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f44736-7e54-4209-9ba8-efcbf094dd51 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.373950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f44736-7e54-4209-9ba8-efcbf094dd51 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:49.374462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f44736-7e54-4209-9ba8-efcbf094dd51 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:49.376674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f44736-7e54-4209-9ba8-efcbf094dd51 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:49.392054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 364/734] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:49.393577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5c37a-35fe-49b0-9424-f7d4244cc6ed None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.394216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5c37a-35fe-49b0-9424-f7d4244cc6ed None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.394789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5c37a-35fe-49b0-9424-f7d4244cc6ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.408886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef29fbd-613e-4fa3-b202-bccc4d3c65c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:49.409822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef29fbd-613e-4fa3-b202-bccc4d3c65c0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.409822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef29fbd-613e-4fa3-b202-bccc4d3c65c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.410114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef29fbd-613e-4fa3-b202-bccc4d3c65c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.412666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:49.417308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e3c5c37a-35fe-49b0-9424-f7d4244cc6ed None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:49.418851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e3c5c37a-35fe-49b0-9424-f7d4244cc6ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:49.480867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef29fbd-613e-4fa3-b202-bccc4d3c65c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:49.482363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef29fbd-613e-4fa3-b202-bccc4d3c65c0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:49.495666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 365/735] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:49.496843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c5c37a-35fe-49b0-9424-f7d4244cc6ed None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:49.573505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:49.610379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c5c37a-35fe-49b0-9424-f7d4244cc6ed None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:49.649428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20f9247-8e6b-4dd7-8f04-4ab5735929d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:49.650462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f9247-8e6b-4dd7-8f04-4ab5735929d4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.650979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f9247-8e6b-4dd7-8f04-4ab5735929d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.651442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f9247-8e6b-4dd7-8f04-4ab5735929d4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.659546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20f9247-8e6b-4dd7-8f04-4ab5735929d4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:49.661245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20f9247-8e6b-4dd7-8f04-4ab5735929d4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:49.666368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 371/736] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:49.671410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef29fbd-613e-4fa3-b202-bccc4d3c65c0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c20f9247-8e6b-4dd7-8f04-4ab5735929d4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:49.674530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:49.675490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:49.676470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:49.678988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 366/737] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:49 2022] GET /identity/v3/users/swift => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:49.685482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8be9d-d9c4-4907-aad1-c4258bb55953 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.686127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8be9d-d9c4-4907-aad1-c4258bb55953 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.686593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f8be9d-d9c4-4907-aad1-c4258bb55953 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.688318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 367/738] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:49.689210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8be9d-d9c4-4907-aad1-c4258bb55953 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:49.705922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:49.773378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef29fbd-613e-4fa3-b202-bccc4d3c65c0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:49.804496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f8be9d-d9c4-4907-aad1-c4258bb55953 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:49.809892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10ac775-c13c-45fd-984d-3dd099167411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:49.810524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10ac775-c13c-45fd-984d-3dd099167411 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.811806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10ac775-c13c-45fd-984d-3dd099167411 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.812091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10ac775-c13c-45fd-984d-3dd099167411 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.823124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10ac775-c13c-45fd-984d-3dd099167411 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:49.824686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10ac775-c13c-45fd-984d-3dd099167411 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:49.830356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:49.839661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae40bc07-b317-4db6-bd20-17476fca1774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:49.840271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae40bc07-b317-4db6-bd20-17476fca1774 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.842405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae40bc07-b317-4db6-bd20-17476fca1774 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.842405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae40bc07-b317-4db6-bd20-17476fca1774 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-a10ac775-c13c-45fd-984d-3dd099167411 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:49.842598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:49.843527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:49.844446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:49.844446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:49.844446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:49.844620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae40bc07-b317-4db6-bd20-17476fca1774 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:49.844620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae40bc07-b317-4db6-bd20-17476fca1774 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:49.844760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 372/739] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:49.844829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae40bc07-b317-4db6-bd20-17476fca1774 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:49.844897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c09b7dd-aca3-4967-9940-942007b38b20 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.844897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c09b7dd-aca3-4967-9940-942007b38b20 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.844897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c09b7dd-aca3-4967-9940-942007b38b20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.865777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 368/740] 10.222.0.16 () {56 vars in 1135 bytes} [Thu Aug 25 04:53:49 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:49.865777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0b4ab-f0ed-4f83-a949-acd979d69242 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:49.865777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0b4ab-f0ed-4f83-a949-acd979d69242 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:49.865777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0b4ab-f0ed-4f83-a949-acd979d69242 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:49.865777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 369/741] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:49.865777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e0b4ab-f0ed-4f83-a949-acd979d69242 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:49.871964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-9c09b7dd-aca3-4967-9940-942007b38b20 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:49.873172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9c09b7dd-aca3-4967-9940-942007b38b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:49.938513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 373/742] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:49.939258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c09b7dd-aca3-4967-9940-942007b38b20 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:49.956847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:49.968355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e0b4ab-f0ed-4f83-a949-acd979d69242 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.004335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456ec9cd-0f89-468a-a520-e07923905f40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.004934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456ec9cd-0f89-468a-a520-e07923905f40 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.005209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456ec9cd-0f89-468a-a520-e07923905f40 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.005484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456ec9cd-0f89-468a-a520-e07923905f40 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.006158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456ec9cd-0f89-468a-a520-e07923905f40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:50.006462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456ec9cd-0f89-468a-a520-e07923905f40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:50.007810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456ec9cd-0f89-468a-a520-e07923905f40 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:50.019880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 370/743] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:50.021051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1db7a0-2fed-441f-bc44-e211da8829b1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.021339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1db7a0-2fed-441f-bc44-e211da8829b1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.021640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1db7a0-2fed-441f-bc44-e211da8829b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.039068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9c1db7a0-2fed-441f-bc44-e211da8829b1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:50.040786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9c1db7a0-2fed-441f-bc44-e211da8829b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:50.049649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c09b7dd-aca3-4967-9940-942007b38b20 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.083761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:50.085787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23ee64-8415-4715-a878-62e1d7ffa0a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.087211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23ee64-8415-4715-a878-62e1d7ffa0a2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.087496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23ee64-8415-4715-a878-62e1d7ffa0a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.087773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c23ee64-8415-4715-a878-62e1d7ffa0a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.111544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 371/744] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 25 04:53:50.112707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1db7a0-2fed-441f-bc44-e211da8829b1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:50.167114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c23ee64-8415-4715-a878-62e1d7ffa0a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:50.169071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c23ee64-8415-4715-a878-62e1d7ffa0a2 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:50.186691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:50.251054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1db7a0-2fed-441f-bc44-e211da8829b1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.283761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 374/745] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 25 04:53:50.284797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23ee64-8415-4715-a878-62e1d7ffa0a2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:50.290966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72402ad3-871b-43a9-aa55-39d2b876de21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.291573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72402ad3-871b-43a9-aa55-39d2b876de21 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.291848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72402ad3-871b-43a9-aa55-39d2b876de21 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.292116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72402ad3-871b-43a9-aa55-39d2b876de21 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.292754 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72402ad3-871b-43a9-aa55-39d2b876de21 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:50.294152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72402ad3-871b-43a9-aa55-39d2b876de21 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:50.387537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c23ee64-8415-4715-a878-62e1d7ffa0a2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.403131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:50.410112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 372/746] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:53:50 2022] POST /identity/v3/services => generated 240 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:50.410867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72402ad3-871b-43a9-aa55-39d2b876de21 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:50.427621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.428260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.428537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.428805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.437072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:50.438895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:50.451714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:50.452835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:50.454115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 375/747] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:50.454748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:50.517939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:50.567204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb04047-e4cb-49a8-b9fc-c17021a8b373 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.589264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72402ad3-871b-43a9-aa55-39d2b876de21 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.606823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.607879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.608610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.609040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.623888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:50.625515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:50.626228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b84441-da39-47f7-8c93-5ae30e33aa73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.626861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b84441-da39-47f7-8c93-5ae30e33aa73 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.627140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b84441-da39-47f7-8c93-5ae30e33aa73 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.627411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b84441-da39-47f7-8c93-5ae30e33aa73 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.638180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b84441-da39-47f7-8c93-5ae30e33aa73 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-11b84441-da39-47f7-8c93-5ae30e33aa73 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:50.642793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:50.643899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 373/748] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:50.645174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b84441-da39-47f7-8c93-5ae30e33aa73 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:50.646400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:50.647335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:50.648212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:50.648212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:50.648212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:50.649730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 376/749] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:50.650606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:50.677367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:50.747573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:50.754039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b84441-da39-47f7-8c93-5ae30e33aa73 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.757405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981b4e23-29be-46f8-bf3a-ee3aa7c345ad None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.788599 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3178663a-69e8-4de8-8228-77617ad7914c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.789133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178663a-69e8-4de8-8228-77617ad7914c None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.790040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178663a-69e8-4de8-8228-77617ad7914c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.790289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178663a-69e8-4de8-8228-77617ad7914c None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.790855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3178663a-69e8-4de8-8228-77617ad7914c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:50.792208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3178663a-69e8-4de8-8228-77617ad7914c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:50.794023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.794633 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.794918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.795192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.796062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:50.796740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:50.798419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:50.810129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 377/750] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:50.811017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:50.863946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 374/751] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:53:50 2022] POST /identity/v3/services => generated 251 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:50.864816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3178663a-69e8-4de8-8228-77617ad7914c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:50.911023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:50.930824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4a14f2-cdf5-4a9a-9154-094bc5694ca6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:50.977826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:50.978861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:50.979356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:50.979871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:50.980858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:50.981447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:50.983076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:50.994661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 378/752] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:50.995983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:51.007731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3178663a-69e8-4de8-8228-77617ad7914c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:51.051818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:51.052370 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:51.052652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:51.052926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:51.053583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:51.053910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:51.055351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:51.071172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 375/753] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:51.072032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:51.096258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:51.116805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3c732d-2bf4-4090-a940-a268dde210d2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:51.160254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:51.160867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:51.161458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:51.161765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:51.162779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=b7ba48fe350b4644a9f5edd0729a0536) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:51.163092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:51.165139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:51.172141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 379/754] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:50 2022] GET /identity/v3/role_assignments?user.id=b7ba48fe350b4644a9f5edd0729a0536&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:51.172939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:51.190018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:51.196875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82e94e0-767f-4d1a-92da-e386df396914 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:51.242256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:51.244081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:51.244658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:51.245071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:51.257736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:51.259536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:51.277258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:51.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:51.279573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:51.279573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:51.279573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:51.280703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 376/755] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:51.287119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:51.401164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:51.425339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d56453-24c7-4d9c-b47f-92dea159894e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:51.441613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac603b92-8d03-4bc6-9b5b-d17779dcd542 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:51.496058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:51.496701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:51.496981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:51.497253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:51.501610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:51.502237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:51.502510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:51.502778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:51.503430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:51.503731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:51.505092 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:51.516686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:51.518192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:51.534193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 377/756] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:51.535086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:51.536701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:51.537737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:51.538809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:51.538809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:51.538809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:51.538809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:51.538809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:51.538809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:51.538809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:51.538809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:51.541879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 380/757] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 365 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:51.546109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:51.673388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:51.713474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513361ea-56a4-4383-8b2f-5b60b6be56b2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:51.738634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682a7aa1-3b23-47c4-a1f4-7c430c96db7b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:51.764368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:51.781799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:51.782437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:51.782714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:51.784652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:51.795358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:51.797055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:51.812472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:51.813430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:51.814408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 378/758] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:53:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:51.819038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:51.824700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:51.825329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:51.825619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:51.825902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:51.826604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:51.826915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:51.828779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:51.850974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 381/759] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:51.858588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:51.938280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d72dff1-3f9d-47a9-b664-4d73b0b28f22 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:51.974339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:51.974954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:51.975235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:51.975507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:51.976244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:51.976548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:51.978064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:52.001455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 379/760] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:53:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:52.002822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:52.035007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa216ab-63f8-4030-9122-44e63c76eaa5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:52.079033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:52.088050 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:52.105090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:52.105835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:52.106122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:52.106391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:52.107120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=c77473684c58457aa53f73a9c30319d0) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:52.107429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:52.108867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:52.129885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 382/761] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:51 2022] GET /identity/v3/role_assignments?user.id=c77473684c58457aa53f73a9c30319d0&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 259 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:52.176280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c73041c-d512-4dab-bd4f-c7c8a9d84a5c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:52.214501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:52.215070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:52.215348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:52.215619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:52.227408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:52.230946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:52.246651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:52.247678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:52.248639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:52.248639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:52.248639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:52.252565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 380/762] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:52.253988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:52.375782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bacb1bc-f366-4f87-9efa-05280e6f5105 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:52.422382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b7059d-811b-4558-8ba9-6f16a88b2959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:52.423045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7059d-811b-4558-8ba9-6f16a88b2959 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:52.423324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7059d-811b-4558-8ba9-6f16a88b2959 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:52.423595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7059d-811b-4558-8ba9-6f16a88b2959 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:52.424234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b7059d-811b-4558-8ba9-6f16a88b2959 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:52.424533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b7059d-811b-4558-8ba9-6f16a88b2959 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:52.425929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b7059d-811b-4558-8ba9-6f16a88b2959 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:52.438578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 383/763] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:52.443870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:52.556278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:52.565900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1512bae4-2bc0-4fc8-8884-5c877891266f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:52.589573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade78f58-a51f-479f-9a7a-2699f9e95e91 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:52.589933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade78f58-a51f-479f-9a7a-2699f9e95e91 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:52.590168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade78f58-a51f-479f-9a7a-2699f9e95e91 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:52.591956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 384/764] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:53:52.597919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04386da7-dd23-4b7f-adec-89237c20f940 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:52.598201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04386da7-dd23-4b7f-adec-89237c20f940 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:52.598481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04386da7-dd23-4b7f-adec-89237c20f940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:52.619261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-04386da7-dd23-4b7f-adec-89237c20f940 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:52.620450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-04386da7-dd23-4b7f-adec-89237c20f940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:52.620868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:52.621506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:52.621806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:52.622076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:52.637464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:52.639149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:52.656532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:52.657478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:52.658456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 381/765] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:52.663689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:52.679332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:52.719357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 385/766] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:53:52.800778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43743edd-1e03-4f1c-bf0d-86514bc28623 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:52.837762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:52.838299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:52.838575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:52.838845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:52.839532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:52.839834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:52.841319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:52.854422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 382/767] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:52.860173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04386da7-dd23-4b7f-adec-89237c20f940 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:52.979394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04386da7-dd23-4b7f-adec-89237c20f940 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:53.022190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc497fee-3fd0-432f-9d93-110f3cc181b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:53.022853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc497fee-3fd0-432f-9d93-110f3cc181b8 None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.023130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc497fee-3fd0-432f-9d93-110f3cc181b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.023403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc497fee-3fd0-432f-9d93-110f3cc181b8 None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/eb5ac6ef2f5048d3aed0d7a91abc4cac/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.025336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc497fee-3fd0-432f-9d93-110f3cc181b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=eb5ac6ef2f5048d3aed0d7a91abc4cac, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:53.026954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc497fee-3fd0-432f-9d93-110f3cc181b8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:53.105406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:53.129627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 386/768] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:53:52 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/eb5ac6ef2f5048d3aed0d7a91abc4cac/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:53.137720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:53.155170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b47ef5-973e-49d6-90e9-a8885902f819 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.155807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b47ef5-973e-49d6-90e9-a8885902f819 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.156303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b47ef5-973e-49d6-90e9-a8885902f819 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.158258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 387/769] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:53.163945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4cdeae-92e8-47c2-870b-e4ac4fb7ae17 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.165024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4cdeae-92e8-47c2-870b-e4ac4fb7ae17 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.165577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4cdeae-92e8-47c2-870b-e4ac4fb7ae17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.205423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3d4cdeae-92e8-47c2-870b-e4ac4fb7ae17 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:53.207997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3d4cdeae-92e8-47c2-870b-e4ac4fb7ae17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:53.261944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:53.292398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded7e505-35b6-455b-bddf-87f758f4bcf6 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:53.297781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 388/770] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:53:53.340385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02de3a37-979e-4c8b-a8b5-0697f45e7ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:53.341540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de3a37-979e-4c8b-a8b5-0697f45e7ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.342037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de3a37-979e-4c8b-a8b5-0697f45e7ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.342464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de3a37-979e-4c8b-a8b5-0697f45e7ab4 None admin] PATH_INFO: `/v3/services/network` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.343275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02de3a37-979e-4c8b-a8b5-0697f45e7ab4 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:53.346929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02de3a37-979e-4c8b-a8b5-0697f45e7ab4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-02de3a37-979e-4c8b-a8b5-0697f45e7ab4 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:53.357152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:53.358246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:53.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:53.362138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 383/771] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:53 2022] GET /identity/v3/services/network => generated 88 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:53.368571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4cdeae-92e8-47c2-870b-e4ac4fb7ae17 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:53.371857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f7c3b-53bd-4bc6-ab4b-5db647377922 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.372363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f7c3b-53bd-4bc6-ab4b-5db647377922 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.372831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f7c3b-53bd-4bc6-ab4b-5db647377922 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.376071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 384/772] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:53.381483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c7aaa-b2b0-451c-9ba3-341e29a7c616 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.381483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c7aaa-b2b0-451c-9ba3-341e29a7c616 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.381483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5c7aaa-b2b0-451c-9ba3-341e29a7c616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.406740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8b5c7aaa-b2b0-451c-9ba3-341e29a7c616 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:53.407954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8b5c7aaa-b2b0-451c-9ba3-341e29a7c616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:53.509630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 385/773] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:53:53.525858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4cdeae-92e8-47c2-870b-e4ac4fb7ae17 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:53.574337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:53.575291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.575766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.576258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.577075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] RBAC: Adding query filter params (name=network) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:53.577571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:53.579160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:53.590201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 389/774] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:53:53 2022] GET /identity/v3/services?name=network => generated 124 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:53.595920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5c7aaa-b2b0-451c-9ba3-341e29a7c616 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:53.621092 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:53.631501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:53.723401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5c7aaa-b2b0-451c-9ba3-341e29a7c616 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:53.728330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:53.762821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:53.763437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.763720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.763995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.764629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] RBAC: Adding query filter params (type=network) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:53.764934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:53.766315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:53.772072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 386/775] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:53:53 2022] GET /identity/v3/services?type=network => generated 350 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:53.782762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:53.827083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb23d5d3-eaab-47d1-9100-01f8fd7a6dda None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:53.868408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:53.869053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.869338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.869650 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.873120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:53.874536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:53.887085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:53.888078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:53.888925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 390/776] 10.222.0.16 () {56 vars in 1129 bytes} [Thu Aug 25 04:53:53 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:53.889761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:53.912780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:53.942599 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84158006-9a61-4ec8-ab85-16b7df8fd98a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:53.988398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:53.989016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:53.989295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:53.989566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:53.990301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] RBAC: Adding query filter params (interface=public, service_id=a399c15e77b542e0970e536ce598e2c6, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:53.990611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:53.992044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:54.003303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 387/777] 10.222.0.16 () {56 vars in 1251 bytes} [Thu Aug 25 04:53:53 2022] GET /identity/v3/endpoints?service_id=a399c15e77b542e0970e536ce598e2c6&interface=public®ion_id=RegionOne => generated 194 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:54.004205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:54.024334 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:54.030377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d694d9d-9e67-443f-aed5-561d1e50edd6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:54.082058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4683986-ea51-427d-9230-c2e32f4279a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:54.083040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4683986-ea51-427d-9230-c2e32f4279a0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.083543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4683986-ea51-427d-9230-c2e32f4279a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.084233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4683986-ea51-427d-9230-c2e32f4279a0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.099105 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4683986-ea51-427d-9230-c2e32f4279a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-f4683986-ea51-427d-9230-c2e32f4279a0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:54.106383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:54.107323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:54.108244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:54.115636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 391/778] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:54.115636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4683986-ea51-427d-9230-c2e32f4279a0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:54.133303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87a867b-37eb-41c6-aeac-c28bf8a253d4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:54.181847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:54.183075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.183963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.184726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.185557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:54.186075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:54.188692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:54.200540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 388/779] 10.222.0.16 () {56 vars in 1124 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:54.201722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:54.207086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:54.236518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4683986-ea51-427d-9230-c2e32f4279a0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:54.282839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4c6ff3-37a1-4de6-81dc-b6fdcfa01717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:54.284072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4c6ff3-37a1-4de6-81dc-b6fdcfa01717 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.284951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4c6ff3-37a1-4de6-81dc-b6fdcfa01717 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.285712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4c6ff3-37a1-4de6-81dc-b6fdcfa01717 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.286780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4c6ff3-37a1-4de6-81dc-b6fdcfa01717 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:54.289194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4c6ff3-37a1-4de6-81dc-b6fdcfa01717 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:54.297142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 392/780] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/services => generated 772 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:54.298591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4c6ff3-37a1-4de6-81dc-b6fdcfa01717 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:54.334142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ee6bfc-bf46-4477-adf9-3bd7c7ead2e4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:54.334968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:54.375839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:54.376442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.376726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.377000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.377734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:54.378041 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:54.379531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:54.393963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 389/781] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:54.396212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:54.458372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4c6ff3-37a1-4de6-81dc-b6fdcfa01717 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:54.507633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:54.537538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:54.538199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.538463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.538785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.539980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:54.540280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:54.541748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:54.549151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 393/782] 10.222.0.16 () {56 vars in 1124 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:54.556127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:54.561569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabea41d-ddd0-4a93-bf3e-b4560573a025 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:54.610771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be250f27-f38b-4311-a946-0fa3b9302c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:54.612149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be250f27-f38b-4311-a946-0fa3b9302c39 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.612149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be250f27-f38b-4311-a946-0fa3b9302c39 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.612149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be250f27-f38b-4311-a946-0fa3b9302c39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.626591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be250f27-f38b-4311-a946-0fa3b9302c39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:54.628543 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be250f27-f38b-4311-a946-0fa3b9302c39 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-be250f27-f38b-4311-a946-0fa3b9302c39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:54.647295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:54.648513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:54.649656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:54.649656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:54.649656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:54.653678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 390/783] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:54.654932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5294c877-cb68-480d-8715-09d2525b209d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.655220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5294c877-cb68-480d-8715-09d2525b209d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.655499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5294c877-cb68-480d-8715-09d2525b209d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.656763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 391/784] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:54.657450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5294c877-cb68-480d-8715-09d2525b209d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:54.689437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3bdea3-6a78-4d16-b767-84f7e20eb34f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:54.721962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:54.730641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1c13d4-90bf-43dc-b80b-4e8f0518299f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:54.731784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1c13d4-90bf-43dc-b80b-4e8f0518299f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.732077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1c13d4-90bf-43dc-b80b-4e8f0518299f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.732359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1c13d4-90bf-43dc-b80b-4e8f0518299f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.733250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1c13d4-90bf-43dc-b80b-4e8f0518299f None admin] RBAC: Adding query filter params (interface=public, service_id=226757efb91144e3bcbbf439b2beab90, region_id=RegionOne) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:54.733637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1c13d4-90bf-43dc-b80b-4e8f0518299f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:54.735627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1c13d4-90bf-43dc-b80b-4e8f0518299f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:54.745875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 394/785] 10.222.0.16 () {56 vars in 1251 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/endpoints?service_id=226757efb91144e3bcbbf439b2beab90&interface=public®ion_id=RegionOne => generated 194 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:54.747425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b663d6-751a-4b4b-8ab5-1425cf6579c7 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.747719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b663d6-751a-4b4b-8ab5-1425cf6579c7 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.747998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b663d6-751a-4b4b-8ab5-1425cf6579c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.772412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-22b663d6-751a-4b4b-8ab5-1425cf6579c7 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:54.774233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-22b663d6-751a-4b4b-8ab5-1425cf6579c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:54.795515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5294c877-cb68-480d-8715-09d2525b209d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:54.818505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:54.841187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:54.842081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:54.842365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:54.842664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:54.843321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:54.843763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:54.845152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:54.857623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 395/786] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:53:54.857623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b663d6-751a-4b4b-8ab5-1425cf6579c7 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:54.862565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 392/787] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:54.869012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:55.012302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd105e5-29b1-4e42-b9d0-af439c6d430b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:55.014461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:55.049328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:55.050815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.051098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.051376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.065621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b663d6-751a-4b4b-8ab5-1425cf6579c7 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:55.068343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:55.069815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:55.097008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:55.098055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:53:55.098999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:55.101018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 393/788] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:55.105887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:55.135209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c79d095-cb58-48c1-bcfb-760f8185087f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:55.135824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c79d095-cb58-48c1-bcfb-760f8185087f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.136105 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c79d095-cb58-48c1-bcfb-760f8185087f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.136381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c79d095-cb58-48c1-bcfb-760f8185087f None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.136989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c79d095-cb58-48c1-bcfb-760f8185087f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:55.138401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c79d095-cb58-48c1-bcfb-760f8185087f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:55.151667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 396/789] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:53:54 2022] GET /identity/v3/services => generated 772 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:55.169991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:55.191661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c79d095-cb58-48c1-bcfb-760f8185087f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:55.277678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cab29a-77ec-4559-9a76-08322da36f2e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:55.348214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:55.349223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.349737 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.350215 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.352355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:55.353042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:55.356212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:55.385686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 394/790] 10.222.0.16 () {56 vars in 1149 bytes} [Thu Aug 25 04:53:55 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:55.390906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:55.401644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c79d095-cb58-48c1-bcfb-760f8185087f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:55.451815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcb2ecf-2b29-407f-bf1b-43989eced778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:55.452706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb2ecf-2b29-407f-bf1b-43989eced778 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.452991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb2ecf-2b29-407f-bf1b-43989eced778 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.453266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb2ecf-2b29-407f-bf1b-43989eced778 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.469441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcb2ecf-2b29-407f-bf1b-43989eced778 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-3fcb2ecf-2b29-407f-bf1b-43989eced778 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:55.477932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:55.479066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:55.480124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 397/791] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:55.491349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcb2ecf-2b29-407f-bf1b-43989eced778 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:55.500313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:55.532622 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:55.540720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e867e2-e23d-438f-a3f8-d0e4a0c55169 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:55.580097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e05c25-747e-486c-813a-ada2c068df13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:55.581174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e05c25-747e-486c-813a-ada2c068df13 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.581679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e05c25-747e-486c-813a-ada2c068df13 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.582170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e05c25-747e-486c-813a-ada2c068df13 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.593808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e05c25-747e-486c-813a-ada2c068df13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:55.595542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e05c25-747e-486c-813a-ada2c068df13 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-83e05c25-747e-486c-813a-ada2c068df13 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:55.612188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:55.613211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:55.614491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:55.614491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:55.614491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:55.616449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 395/792] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:55.618154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32faa189-4480-4e50-b08c-3b81f384ba8b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.618656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32faa189-4480-4e50-b08c-3b81f384ba8b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.619169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32faa189-4480-4e50-b08c-3b81f384ba8b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.620820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 396/793] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:55.625745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32faa189-4480-4e50-b08c-3b81f384ba8b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:55.627685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcb2ecf-2b29-407f-bf1b-43989eced778 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:55.677145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6005b200-d5cf-47f4-a7ab-728accb6480b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:55.677786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6005b200-d5cf-47f4-a7ab-728accb6480b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.678068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6005b200-d5cf-47f4-a7ab-728accb6480b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.678399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6005b200-d5cf-47f4-a7ab-728accb6480b None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.679326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6005b200-d5cf-47f4-a7ab-728accb6480b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:55.680183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6005b200-d5cf-47f4-a7ab-728accb6480b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:55.682348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6005b200-d5cf-47f4-a7ab-728accb6480b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:55.697744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 398/794] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:55.699325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c6cd14-4977-4de0-8344-6013168ccb86 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.700459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c6cd14-4977-4de0-8344-6013168ccb86 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.700733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c6cd14-4977-4de0-8344-6013168ccb86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.718930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-40c6cd14-4977-4de0-8344-6013168ccb86 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:55.720122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-40c6cd14-4977-4de0-8344-6013168ccb86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:55.752357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32faa189-4480-4e50-b08c-3b81f384ba8b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:55.784032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:55.792321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:55.793230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:55.793798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:55.794302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:55.795154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:55.795683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:55.797258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:55.797924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 399/795] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:55 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:53:55.804316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c6cd14-4977-4de0-8344-6013168ccb86 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:55.811475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 397/796] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:55.817008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:55.838596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:56.003981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c6cd14-4977-4de0-8344-6013168ccb86 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.012353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff32a28-53d6-44e9-a7fb-026b34020ebb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.044100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.045082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.045566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.046178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.058906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:56.060686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:56.076811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:56.077832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:56.078733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:56.078733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:56.078733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:56.080776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 400/797] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:56.086431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.106330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.107010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.107291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.107557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.116663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:56.118682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:56.140165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:56.153822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:56.154757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:56.155651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:56.155651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:56.155651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:56.155651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:56.155651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:56.155651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:56.155651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:56.155651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:56.156570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 398/798] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:56.156952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.199271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbfb10d-1307-468b-960b-5c9e560fcd15 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.247492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.248459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.248947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.249448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.250040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:56.250884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:56.251412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:56.252985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:56.270785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 401/799] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:56.272026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.274972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e005cd-b743-4a80-8943-a3989deec28f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.313809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657641bb-57d1-4e4c-a595-ad6a7933b6ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.314920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657641bb-57d1-4e4c-a595-ad6a7933b6ff None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.315500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657641bb-57d1-4e4c-a595-ad6a7933b6ff None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.316034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657641bb-57d1-4e4c-a595-ad6a7933b6ff None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.331167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657641bb-57d1-4e4c-a595-ad6a7933b6ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-657641bb-57d1-4e4c-a595-ad6a7933b6ff None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:56.336643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:56.338103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 399/800] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:56.338953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657641bb-57d1-4e4c-a595-ad6a7933b6ff None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.401987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610ce444-2cd4-4347-a301-89f37dee7c07 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.412055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:56.440084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.441492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.441899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.442321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.443127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:56.443536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:56.445081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:56.449599 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657641bb-57d1-4e4c-a595-ad6a7933b6ff None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.457885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 402/801] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:56.458826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.484563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:56.492864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0023b7ba-be32-4361-a8df-1a449b97486c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.493554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023b7ba-be32-4361-a8df-1a449b97486c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.496628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023b7ba-be32-4361-a8df-1a449b97486c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.496628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023b7ba-be32-4361-a8df-1a449b97486c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.501417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0023b7ba-be32-4361-a8df-1a449b97486c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:56.502934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0023b7ba-be32-4361-a8df-1a449b97486c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0023b7ba-be32-4361-a8df-1a449b97486c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:56.515022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:56.516189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:53:56.517428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:56.523189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 400/802] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:53:56 2022] GET /identity/v3/users/swift => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:56.523189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb30eb-bcc9-43e3-88f2-b75a16639f24 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.523189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb30eb-bcc9-43e3-88f2-b75a16639f24 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.523189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bb30eb-bcc9-43e3-88f2-b75a16639f24 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.523189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 401/803] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:56 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:56.523189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bb30eb-bcc9-43e3-88f2-b75a16639f24 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.574528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eff89-78a2-41bd-ab2a-7341c64d5d78 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.609092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.609741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.610026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.610310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.610975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:56.611281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:56.612645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:56.621803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 403/804] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:53:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:56.626974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.633910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bb30eb-bcc9-43e3-88f2-b75a16639f24 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.657295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:56.668765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f3f9e4-6e98-40ee-a880-967e7cd13d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.672735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f3f9e4-6e98-40ee-a880-967e7cd13d43 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.672735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f3f9e4-6e98-40ee-a880-967e7cd13d43 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.672735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f3f9e4-6e98-40ee-a880-967e7cd13d43 None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/b7ba48fe350b4644a9f5edd0729a0536/roles/c9f7a48328df4a7ea7a5981959f395f5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.685398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f3f9e4-6e98-40ee-a880-967e7cd13d43 None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=b7ba48fe350b4644a9f5edd0729a0536, role_id=c9f7a48328df4a7ea7a5981959f395f5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:56.686864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f3f9e4-6e98-40ee-a880-967e7cd13d43 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:56.729302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3df021a-9a83-448a-a548-524887fa63b7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.769622 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 402/805] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:53:56 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/b7ba48fe350b4644a9f5edd0729a0536/roles/c9f7a48328df4a7ea7a5981959f395f5 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:56.769622 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecdf28-cb91-4473-8e93-811c788e1767 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.769622 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecdf28-cb91-4473-8e93-811c788e1767 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.769622 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecdf28-cb91-4473-8e93-811c788e1767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.775421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.776246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.776540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.776831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:56.777614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:56.777937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:56.779489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:56.793796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-42ecdf28-cb91-4473-8e93-811c788e1767 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:56.795115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-42ecdf28-cb91-4473-8e93-811c788e1767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:56.800746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 404/806] 10.222.0.16 () {56 vars in 1135 bytes} [Thu Aug 25 04:53:56 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:56.801799 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.838362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:56.889609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 403/807] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:56 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:53:56.891351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ecdf28-cb91-4473-8e93-811c788e1767 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:56.935364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:56.950100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b345c42-0b81-4325-8a69-f59c29c35951 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:56.992176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:56.993165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:56.993723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:56.994263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.005932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.008484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.017010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ecdf28-cb91-4473-8e93-811c788e1767 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:57.024814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:57.025810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:57.026726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:57.026726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:57.026726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:57.026726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 405/808] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:57.035790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.073635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.075435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.075711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.075980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.100169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.102302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:57.126670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:57.127666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:57.128679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:57.128679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:57.128679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:57.128679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 404/809] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:57.135218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.166560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:57.211318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4d6267-48bb-4ff2-b7f4-8f3a8719c134 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.259977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.260834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.261220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.261616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.262369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:57.262778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.264250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.269106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcff44e-66d7-47bc-987e-1411a1501a84 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.273696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:57.277720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 406/810] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:57.278744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.314595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.316331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.316763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.317178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.318022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:57.318467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.319962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.337854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 405/811] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:57.339351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.396269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8864e-7b00-4752-a8a4-d8b64f7cda72 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.430957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.431583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.431855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.432128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] PATH_INFO: `/v3/services/network` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.432757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.434117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.439199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:57.443224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:57.444244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:57.445121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 407/812] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/services/network => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:57.446084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.462847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c0c1fa-fdf8-4f7e-975a-acbd681749e9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.499967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db267aa-0513-4abb-a82c-641473d1037d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.500595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db267aa-0513-4abb-a82c-641473d1037d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.500871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db267aa-0513-4abb-a82c-641473d1037d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.501139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db267aa-0513-4abb-a82c-641473d1037d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.509521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db267aa-0513-4abb-a82c-641473d1037d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.510979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db267aa-0513-4abb-a82c-641473d1037d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.517300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-4db267aa-0513-4abb-a82c-641473d1037d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:53:57.524234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:57.525368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:57.526391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 406/813] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:57.526993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265bc26b-75b0-4313-9ddd-d71a13f7a5f4 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.527072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265bc26b-75b0-4313-9ddd-d71a13f7a5f4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.527375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265bc26b-75b0-4313-9ddd-d71a13f7a5f4 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.532589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 407/814] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:57 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:57.532589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265bc26b-75b0-4313-9ddd-d71a13f7a5f4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.564487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f12b659-a3ef-4516-85f7-46fb109f9752 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.601143 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.602005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.602386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.602767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.609368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.610849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:57.622300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:57.623362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:57.624285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 408/815] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:57.624774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.644922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265bc26b-75b0-4313-9ddd-d71a13f7a5f4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.680799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9e16ae-7dae-4742-bc1c-374a8644b0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.681454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e16ae-7dae-4742-bc1c-374a8644b0f0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.681832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e16ae-7dae-4742-bc1c-374a8644b0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.682105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e16ae-7dae-4742-bc1c-374a8644b0f0 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.682725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9e16ae-7dae-4742-bc1c-374a8644b0f0 None admin] RBAC: Adding query filter params (name=network) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:57.683022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9e16ae-7dae-4742-bc1c-374a8644b0f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.684367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9e16ae-7dae-4742-bc1c-374a8644b0f0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.689140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 408/816] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/services?name=network => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:57.690277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41751004-7a39-41f7-9c0c-6f031bb0f79d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.690554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41751004-7a39-41f7-9c0c-6f031bb0f79d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.690821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41751004-7a39-41f7-9c0c-6f031bb0f79d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.698335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:57.709133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-41751004-7a39-41f7-9c0c-6f031bb0f79d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:57.710354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-41751004-7a39-41f7-9c0c-6f031bb0f79d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:57.740124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73385e33-a237-4d70-89ac-06841e96c187 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.764291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:57.775802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.776417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.776703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.776988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.777749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:57.778059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.779498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.793620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 409/817] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:57 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:53:57.793620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41751004-7a39-41f7-9c0c-6f031bb0f79d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.805347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 409/818] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:57.806363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.917361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41751004-7a39-41f7-9c0c-6f031bb0f79d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.921749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8275fd-4934-494b-936d-94375e565a79 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:57.957740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.958319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.958610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.958889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.959523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] RBAC: Adding query filter params (type=network) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:57.959828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.962015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:57.962151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.962813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:57.963363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:57.963735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:57.964826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:57.965414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:57.966881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:57.967443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 410/819] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/services?type=network => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:57.968422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:57.968779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:57.982584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 410/820] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:57.983466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:58.028198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:58.085375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6080042-a523-49fc-80bc-7ef4013e5b3e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:58.100850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a35024-b4ab-41c8-b61f-c10732f8d8f7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:58.119627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adabd51-0c61-4810-b68a-1f429d033c2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:58.120235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adabd51-0c61-4810-b68a-1f429d033c2c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.120517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adabd51-0c61-4810-b68a-1f429d033c2c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.120792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adabd51-0c61-4810-b68a-1f429d033c2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.136344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc87d329-044e-458f-b19e-4bbb7ad809f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:58.136951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87d329-044e-458f-b19e-4bbb7ad809f4 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.137228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87d329-044e-458f-b19e-4bbb7ad809f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.137500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87d329-044e-458f-b19e-4bbb7ad809f4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.138182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc87d329-044e-458f-b19e-4bbb7ad809f4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:58.139536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc87d329-044e-458f-b19e-4bbb7ad809f4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:58.188406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adabd51-0c61-4810-b68a-1f429d033c2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:58.189885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adabd51-0c61-4810-b68a-1f429d033c2c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:58.249683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 411/821] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:53:57 2022] POST /identity/v3/endpoints => generated 337 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:53:58.249683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc87d329-044e-458f-b19e-4bbb7ad809f4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:58.251114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:58.301805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:58.313279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 411/822] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:53:57 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:53:58.314701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adabd51-0c61-4810-b68a-1f429d033c2c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:58.373653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc87d329-044e-458f-b19e-4bbb7ad809f4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:58.412198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486641e3-d25c-4b8e-a036-ace7a7f185a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:58.412881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486641e3-d25c-4b8e-a036-ace7a7f185a0 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.413166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486641e3-d25c-4b8e-a036-ace7a7f185a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.413441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486641e3-d25c-4b8e-a036-ace7a7f185a0 None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/c77473684c58457aa53f73a9c30319d0/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.416207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486641e3-d25c-4b8e-a036-ace7a7f185a0 None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=c77473684c58457aa53f73a9c30319d0, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:58.417698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486641e3-d25c-4b8e-a036-ace7a7f185a0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:58.435705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adabd51-0c61-4810-b68a-1f429d033c2c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:58.479380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:58.479973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.480254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.480528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.481114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:58.482575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:58.492583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:58.493665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:58.495084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 412/823] 10.222.0.16 () {56 vars in 1129 bytes} [Thu Aug 25 04:53:58 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:58.496041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:58.510118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:58.514636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 412/824] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:53:58 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/c77473684c58457aa53f73a9c30319d0/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:53:58.516113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486641e3-d25c-4b8e-a036-ace7a7f185a0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:58.627416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85cc00-5159-4e79-bb87-333d4d0c4bcf None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:58.646006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486641e3-d25c-4b8e-a036-ace7a7f185a0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:58.655346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:58.675657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:58.677032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.677737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.678559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.689281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:58.689873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:58.689989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.690190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.690444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.691074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:58.691344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:58.691449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:58.692698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:58.697634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 413/825] 10.222.0.16 () {56 vars in 1124 bytes} [Thu Aug 25 04:53:58 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:58.702963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:58.708157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:58.709122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:58.710111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:58.710111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:58.710111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:58.713710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 413/826] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:58.715975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:58.791758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:58.827034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddac16a-f031-4353-a927-b3bcf1a7f890 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:58.840106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf055fd-a0c5-494f-9e26-12452a6b6874 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:58.868405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d253bfc4-70db-449a-8379-545bb606b24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:58.869026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d253bfc4-70db-449a-8379-545bb606b24d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.869311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d253bfc4-70db-449a-8379-545bb606b24d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.869607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d253bfc4-70db-449a-8379-545bb606b24d None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.870255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d253bfc4-70db-449a-8379-545bb606b24d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:58.870560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d253bfc4-70db-449a-8379-545bb606b24d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:58.872047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d253bfc4-70db-449a-8379-545bb606b24d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:58.876928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 414/827] 10.222.0.16 () {56 vars in 1124 bytes} [Thu Aug 25 04:53:58 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:58.878150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446901ea-d3e9-4a97-9508-66aebb8a2d70 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.878402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446901ea-d3e9-4a97-9508-66aebb8a2d70 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.878679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446901ea-d3e9-4a97-9508-66aebb8a2d70 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.879985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 415/828] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:53:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:53:58.882217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446901ea-d3e9-4a97-9508-66aebb8a2d70 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:58.888413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebeac046-f491-46d6-8e2a-49b10621ec76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:58.889038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeac046-f491-46d6-8e2a-49b10621ec76 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.889324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeac046-f491-46d6-8e2a-49b10621ec76 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.889620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebeac046-f491-46d6-8e2a-49b10621ec76 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.890278 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeac046-f491-46d6-8e2a-49b10621ec76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:58.890589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeac046-f491-46d6-8e2a-49b10621ec76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:58.891963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebeac046-f491-46d6-8e2a-49b10621ec76 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:58.905070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 414/829] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:58.906181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6010ac-aca3-4a89-9b94-e5415abded6d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:58.906465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6010ac-aca3-4a89-9b94-e5415abded6d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:58.906736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6010ac-aca3-4a89-9b94-e5415abded6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:58.922564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:58.924922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-dc6010ac-aca3-4a89-9b94-e5415abded6d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:53:58.926123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-dc6010ac-aca3-4a89-9b94-e5415abded6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:53:58.996832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 415/830] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:53:58 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:53:58.998700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446901ea-d3e9-4a97-9508-66aebb8a2d70 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:59.002479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6010ac-aca3-4a89-9b94-e5415abded6d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:59.041955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5bd7f-ec98-4613-a3af-447960e653a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:59.042737 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5bd7f-ec98-4613-a3af-447960e653a0 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:59.043908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5bd7f-ec98-4613-a3af-447960e653a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:59.044196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d5bd7f-ec98-4613-a3af-447960e653a0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:59.044874 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5bd7f-ec98-4613-a3af-447960e653a0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:59.046264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d5bd7f-ec98-4613-a3af-447960e653a0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:59.063215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:59.109980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6010ac-aca3-4a89-9b94-e5415abded6d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:59.127654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 416/831] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:53:58 2022] POST /identity/v3/endpoints => generated 346 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:53:59.160506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:59.161553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:59.162184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:59.162668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:59.171816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:59.173839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:53:59.188512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:59.189521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:59.190500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:53:59.190500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:53:59.190500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:53:59.190500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:53:59.190500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:53:59.190500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:53:59.190500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:53:59.190500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:59.193065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 416/832] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:53:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:59.198760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5bd7f-ec98-4613-a3af-447960e653a0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:59.262862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:59.311753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:59.322556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d5bd7f-ec98-4613-a3af-447960e653a0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:59.365026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:59.366050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:59.366511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:59.367021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:59.367745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:59.368067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:59.369537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:59.384741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 417/833] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:53:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:59.389059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:59.400932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea438d8-06f8-48d2-b728-219b5ab3f329 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:59.413953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:59.450334 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fc91d4-0e27-443c-b263-0cb78f3dfba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:59.451070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc91d4-0e27-443c-b263-0cb78f3dfba6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:59.451294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc91d4-0e27-443c-b263-0cb78f3dfba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:59.451569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc91d4-0e27-443c-b263-0cb78f3dfba6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:59.470484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fc91d4-0e27-443c-b263-0cb78f3dfba6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-49fc91d4-0e27-443c-b263-0cb78f3dfba6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:53:59.477527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:59.478533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:53:59.479385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 417/834] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:53:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:53:59.485765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fc91d4-0e27-443c-b263-0cb78f3dfba6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:59.499711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2153bb1-4972-4aaa-b8d3-f455ef76179d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:59.539952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:59.540659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:59.541122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:59.542170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:59.543372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:59.543852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:59.546032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:59.561695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 418/835] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:53:59 2022] GET /identity/v3/role_assignments?user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:53:59.612864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fc91d4-0e27-443c-b263-0cb78f3dfba6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:59.652280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:59.652910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:59.653191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:59.653462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:59.654172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:53:59.654476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:59.655878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:59.666173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 418/836] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:53:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:53:59.677952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:59.706030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:59.789614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4deac02-75e0-40d3-88be-5c7d640f1e07 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:53:59.825231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:53:59.825890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:53:59.826169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:53:59.826443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:53:59.837375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:53:59.838843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:53:59.854479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:53:59.855465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:53:59.856382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:53:59.856382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:53:59.856382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:53:59.856382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 419/837] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:53:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:53:59.861395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:53:59.894135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:53:59.965402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa01cc8f-05c7-4f8d-acb9-95cef1c0de41 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:00.000040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:00.000702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:00.000978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:00.001253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:00.001908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:00.002204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:00.003556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:00.015398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 419/838] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:53:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:00.020959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:00.090923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:00.127503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671869eb-a7b9-4e64-87d6-80362459b4b8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:00.162035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:00.162586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:00.162861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:00.163181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:00.170940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:00.172485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:00.184251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:00.185226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:00.186240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 420/839] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:54:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:00.191502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:00.289024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:00.298658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9222744b-3e53-4956-9306-9c63d5bebf77 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:00.339494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0451b8e8-3943-4b6d-a110-8a69c81de18a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:00.340124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0451b8e8-3943-4b6d-a110-8a69c81de18a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:00.340411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0451b8e8-3943-4b6d-a110-8a69c81de18a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:00.340683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0451b8e8-3943-4b6d-a110-8a69c81de18a None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:00.341570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0451b8e8-3943-4b6d-a110-8a69c81de18a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:00.341906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0451b8e8-3943-4b6d-a110-8a69c81de18a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:00.343386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0451b8e8-3943-4b6d-a110-8a69c81de18a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:00.362938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 420/840] 10.222.0.16 () {56 vars in 1149 bytes} [Thu Aug 25 04:54:00 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:00.368694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:00.470040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7877797-d58e-4b06-b000-db831a0fb133 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:00.500359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:00.508915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead19afb-4c5e-4895-bf54-2aab04d8bffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:00.509554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead19afb-4c5e-4895-bf54-2aab04d8bffd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:00.509891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead19afb-4c5e-4895-bf54-2aab04d8bffd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:00.510179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead19afb-4c5e-4895-bf54-2aab04d8bffd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:00.584038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead19afb-4c5e-4895-bf54-2aab04d8bffd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:00.585493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead19afb-4c5e-4895-bf54-2aab04d8bffd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:00.594299 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abcb661-1353-4bce-aae1-9d53e3127fa3 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:00.594758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abcb661-1353-4bce-aae1-9d53e3127fa3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:00.594837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abcb661-1353-4bce-aae1-9d53e3127fa3 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:00.598041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 421/841] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:00.602407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f907b52-b6e2-4e1d-b0e0-7a8edd5b0c7e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:00.602694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f907b52-b6e2-4e1d-b0e0-7a8edd5b0c7e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:00.602967 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f907b52-b6e2-4e1d-b0e0-7a8edd5b0c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:00.625044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6f907b52-b6e2-4e1d-b0e0-7a8edd5b0c7e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:00.626267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6f907b52-b6e2-4e1d-b0e0-7a8edd5b0c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:00.694477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 421/842] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:00 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:54:00.700106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead19afb-4c5e-4895-bf54-2aab04d8bffd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:00.717682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 422/843] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:00 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:00.737222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:00.813986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead19afb-4c5e-4895-bf54-2aab04d8bffd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:00.849444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:00.850105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:00.850383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:00.850660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:00.861320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:00.862797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:00.877582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:00.878570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:00.879485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:00.879485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:00.879485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:00.879485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 422/844] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:00.884337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f907b52-b6e2-4e1d-b0e0-7a8edd5b0c7e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:00.962867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:00.984103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f907b52-b6e2-4e1d-b0e0-7a8edd5b0c7e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.019383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.019976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.020260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.020530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.021163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:01.021459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:01.022847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:01.035271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.035609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 423/845] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:01.041394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.134652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:01.156923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47e148f-9876-4617-ae88-2d0dbb590ed0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.171435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e06223-265d-42b6-adcc-0fa2c6e235cd None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.193547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882441ac-5710-4b5c-a45b-145d1117e35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.194343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882441ac-5710-4b5c-a45b-145d1117e35b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.194717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882441ac-5710-4b5c-a45b-145d1117e35b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.195142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-882441ac-5710-4b5c-a45b-145d1117e35b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.205904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-882441ac-5710-4b5c-a45b-145d1117e35b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-882441ac-5710-4b5c-a45b-145d1117e35b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:01.212194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:01.214054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:01.215368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 423/846] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:01.221613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682b7e30-ea62-4fc1-9df9-776ffa61b66d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.221613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b7e30-ea62-4fc1-9df9-776ffa61b66d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.221613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b7e30-ea62-4fc1-9df9-776ffa61b66d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.221613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682b7e30-ea62-4fc1-9df9-776ffa61b66d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.222310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882441ac-5710-4b5c-a45b-145d1117e35b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.228438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682b7e30-ea62-4fc1-9df9-776ffa61b66d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:01.229953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682b7e30-ea62-4fc1-9df9-776ffa61b66d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:01.231988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-682b7e30-ea62-4fc1-9df9-776ffa61b66d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:01.246823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:01.252081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 424/847] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0049c-82f0-470a-97af-91ddb5e045d3 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.252977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0049c-82f0-470a-97af-91ddb5e045d3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.253575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0049c-82f0-470a-97af-91ddb5e045d3 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.253575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 425/848] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:01.256617 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4266805d-307f-45a9-8f62-ee495ef5476c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.256899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4266805d-307f-45a9-8f62-ee495ef5476c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.257171 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4266805d-307f-45a9-8f62-ee495ef5476c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.275995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4266805d-307f-45a9-8f62-ee495ef5476c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:01.277100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4266805d-307f-45a9-8f62-ee495ef5476c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:01.345971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-882441ac-5710-4b5c-a45b-145d1117e35b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.346166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 426/849] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:01 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:01.346866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4266805d-307f-45a9-8f62-ee495ef5476c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.397940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.400435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.401059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.402062 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.402972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:01.403904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:01.406044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:01.413636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:01.419023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 424/850] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:01.424591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.468908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4266805d-307f-45a9-8f62-ee495ef5476c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.496312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:01.505269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.506178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.511757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.511757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.511757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:01.511757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:01.511757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:01.528118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 427/851] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:01.529276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.539403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66609b84-40b2-4f20-849b-88c8ad919466 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.577456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46c08c3-86ba-46ed-8ab4-5dc3a3ab3cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.578111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c08c3-86ba-46ed-8ab4-5dc3a3ab3cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.578454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c08c3-86ba-46ed-8ab4-5dc3a3ab3cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.578723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46c08c3-86ba-46ed-8ab4-5dc3a3ab3cd4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.590205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46c08c3-86ba-46ed-8ab4-5dc3a3ab3cd4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:01.591746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46c08c3-86ba-46ed-8ab4-5dc3a3ab3cd4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e46c08c3-86ba-46ed-8ab4-5dc3a3ab3cd4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:01.608669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:01.609724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:01.610633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:01.610633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:01.610633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:01.610633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 425/852] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:01.611435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab264b1-926a-4be6-9ab2-6162a11d806d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.611618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab264b1-926a-4be6-9ab2-6162a11d806d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.611845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab264b1-926a-4be6-9ab2-6162a11d806d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.613228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 426/853] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:01 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:01.615624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab264b1-926a-4be6-9ab2-6162a11d806d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.649696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c5fe3-58ad-4838-9ec0-73b23bd38a55 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.670423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:01.683550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.684153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.684451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.684707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.685430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=b7ba48fe350b4644a9f5edd0729a0536) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:01.685778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:01.687212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:01.693093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 428/854] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/role_assignments?user.id=b7ba48fe350b4644a9f5edd0729a0536&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:01.693899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.722377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab264b1-926a-4be6-9ab2-6162a11d806d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.760659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:01.761633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e23fdcb-9b67-4139-91d8-89b4b8c3b192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.762327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23fdcb-9b67-4139-91d8-89b4b8c3b192 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.762692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23fdcb-9b67-4139-91d8-89b4b8c3b192 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.762864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e23fdcb-9b67-4139-91d8-89b4b8c3b192 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.763513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e23fdcb-9b67-4139-91d8-89b4b8c3b192 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:01.763818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e23fdcb-9b67-4139-91d8-89b4b8c3b192 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:01.765502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e23fdcb-9b67-4139-91d8-89b4b8c3b192 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:01.779423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 427/855] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:01.780531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f7f78-ec19-421b-85ca-5a19c0ebc349 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.780888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f7f78-ec19-421b-85ca-5a19c0ebc349 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.781160 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f7f78-ec19-421b-85ca-5a19c0ebc349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.800172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7a8f7f78-ec19-421b-85ca-5a19c0ebc349 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:01.801406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7a8f7f78-ec19-421b-85ca-5a19c0ebc349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:01.821882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a29fc9e-4b8f-4482-ab38-9d412cb3f62d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:01.863621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:01.864272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:01.864550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:01.864821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:01.865409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:01.867350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:01.877823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:01.878818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:01.879685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 429/856] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:01.880485 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.883434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 428/857] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:01 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:01.887602 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8f7f78-ec19-421b-85ca-5a19c0ebc349 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:01.936691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:02.005310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8f7f78-ec19-421b-85ca-5a19c0ebc349 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.012270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc58264-8c87-445f-b13e-e76a24f51c65 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.034722 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:02.044427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.053479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.053479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.053479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.053479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:02.053479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.053479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.054929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 429/858] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:02.056706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.057504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.057948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.058380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.060125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:02.067803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.069447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:02.089383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:02.090635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 25 04:54:02.091833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:02.099281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 430/859] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:54:01 2022] GET /identity/v3/users/swift => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:02.100712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:02.191098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc4d90d-b46f-4de0-8b9d-1b20bca37498 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.199960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:02.215115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5324da09-8ddb-46e4-9316-add72cffffc9 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.237563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.238197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.238472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.238742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.239374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:02.239670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.241024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.245730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 430/860] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:02 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:02.246494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:02.263964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.265013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.265442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.266030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.266792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:02.267103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.268594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.294053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 431/861] 10.222.0.16 () {56 vars in 1135 bytes} [Thu Aug 25 04:54:02 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:02.295346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:02.335984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:02.385018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699e834e-5d9c-4218-bdfe-92bcd686b8dd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.430070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994e7282-313c-4dbd-b598-bd7630a02012 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.433064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.433732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.434009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.434277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.434868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.438079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:02.447885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:02.448884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:02.449852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 431/862] 10.222.0.16 () {56 vars in 1132 bytes} [Thu Aug 25 04:54:02 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:02.458154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:02.475720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fb59f6-d68b-4786-89e0-e927b53b87f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.476886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fb59f6-d68b-4786-89e0-e927b53b87f4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.477230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fb59f6-d68b-4786-89e0-e927b53b87f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.477507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fb59f6-d68b-4786-89e0-e927b53b87f4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.496386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:02.565256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fb59f6-d68b-4786-89e0-e927b53b87f4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.566925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fb59f6-d68b-4786-89e0-e927b53b87f4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.581032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e95bba-e61e-413c-a480-8fd8b88e3fcf None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.613575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:02.629972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.630738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.631026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.631300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.631958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:02.633226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.635044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.641626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 432/863] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:02 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:02.647015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:02.699437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 432/864] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:02 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:54:02.704734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fb59f6-d68b-4786-89e0-e927b53b87f4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:02.767568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:02.774470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971dd1f3-6822-43b8-9dc9-29a63f7c0459 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.823347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.824694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.825731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.826476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.827317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:02.828079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.830077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.837816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 433/865] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:02 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:02.844474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fb59f6-d68b-4786-89e0-e927b53b87f4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:02.889104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:02.889797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:02.890080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:02.890353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:02.892571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:02.903173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:02.904667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:02.921909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:02.923101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:02.924012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:02.924012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:02.924012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:02.927723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 433/866] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:02.929972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:03.047514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b393a815-dce9-46cd-977f-c5a371a6ff53 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:03.082836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:03.083366 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:03.083643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:03.083912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:03.084662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:03.084891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:03.087158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:03.099155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 434/867] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:03.104716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:03.215371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ce4df2-450f-4797-bf03-3e8c985e4eb0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:03.250948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:03.251581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:03.251863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:03.252138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:03.259076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:03.260533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:03.271805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:03.272701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:03.274196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:03.274196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:03.274196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:03.274196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:03.274196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:03.274196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:03.274196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:03.274196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:03.277824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 434/868] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:03.279599 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:03.305753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:03.385539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60403ce-092f-4ffc-8826-2243c110d34e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:03.421680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e751f18-d0bd-4b4a-94e1-5f884219fdc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:03.422648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e751f18-d0bd-4b4a-94e1-5f884219fdc1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:03.423075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e751f18-d0bd-4b4a-94e1-5f884219fdc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:03.423492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e751f18-d0bd-4b4a-94e1-5f884219fdc1 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:03.424383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e751f18-d0bd-4b4a-94e1-5f884219fdc1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:03.424979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e751f18-d0bd-4b4a-94e1-5f884219fdc1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:03.426780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e751f18-d0bd-4b4a-94e1-5f884219fdc1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:03.442222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 435/869] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:54:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:03.448797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:03.502314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:03.557643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa110e7-1f3b-4945-a4a8-7753937a18f5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:03.591378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f19ef17-d53d-4216-adfd-92e19b7c77b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:03.591992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f19ef17-d53d-4216-adfd-92e19b7c77b8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:03.592274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f19ef17-d53d-4216-adfd-92e19b7c77b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:03.592551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f19ef17-d53d-4216-adfd-92e19b7c77b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:03.593270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f19ef17-d53d-4216-adfd-92e19b7c77b8 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=c77473684c58457aa53f73a9c30319d0) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:03.593577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f19ef17-d53d-4216-adfd-92e19b7c77b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:03.595023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f19ef17-d53d-4216-adfd-92e19b7c77b8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:03.605707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 435/870] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:03 2022] GET /identity/v3/role_assignments?user.id=c77473684c58457aa53f73a9c30319d0&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:03.805376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8b513-8309-47c7-835f-7424cd8bc7da None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:03.805743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8b513-8309-47c7-835f-7424cd8bc7da None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:03.805862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef8b513-8309-47c7-835f-7424cd8bc7da None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:03.807425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 436/871] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:03.812640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e90347-c1fd-4e05-aff6-485d0a492932 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:03.813098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e90347-c1fd-4e05-aff6-485d0a492932 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:03.813180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e90347-c1fd-4e05-aff6-485d0a492932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:03.831504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-95e90347-c1fd-4e05-aff6-485d0a492932 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:03.832683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-95e90347-c1fd-4e05-aff6-485d0a492932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:03.904312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 436/872] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:03 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:04.170603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef8b513-8309-47c7-835f-7424cd8bc7da None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:04.254895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:04.278992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef8b513-8309-47c7-835f-7424cd8bc7da None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:04.306804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c1263b-d3d1-47e8-83b3-90cb77b930cf None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.307167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c1263b-d3d1-47e8-83b3-90cb77b930cf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.307504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c1263b-d3d1-47e8-83b3-90cb77b930cf None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.309065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 437/873] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:04.313781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c411788c-974e-4c3d-8854-8bfd19cbede6 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.314066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c411788c-974e-4c3d-8854-8bfd19cbede6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.314348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c411788c-974e-4c3d-8854-8bfd19cbede6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.319205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea0f6e2-331e-4d3e-9bcf-54eaf59d2075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:04.320448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea0f6e2-331e-4d3e-9bcf-54eaf59d2075 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.320729 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea0f6e2-331e-4d3e-9bcf-54eaf59d2075 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.320999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea0f6e2-331e-4d3e-9bcf-54eaf59d2075 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.331779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea0f6e2-331e-4d3e-9bcf-54eaf59d2075 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:04.334913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c411788c-974e-4c3d-8854-8bfd19cbede6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:04.337220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c411788c-974e-4c3d-8854-8bfd19cbede6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-3ea0f6e2-331e-4d3e-9bcf-54eaf59d2075 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:04.337396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:04.342322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:04.343304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 437/874] 10.222.0.16 () {56 vars in 1120 bytes} [Thu Aug 25 04:54:04 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:04.346365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea0f6e2-331e-4d3e-9bcf-54eaf59d2075 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:04.381193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:04.420082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 438/875] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:04 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:04.474185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea0f6e2-331e-4d3e-9bcf-54eaf59d2075 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:04.509004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:04.509632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.509913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.510183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.510844 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:04.511200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:04.512569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:04.521380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 438/876] 10.222.0.16 () {56 vars in 1115 bytes} [Thu Aug 25 04:54:04 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:04.530812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c411788c-974e-4c3d-8854-8bfd19cbede6 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:04.635894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c411788c-974e-4c3d-8854-8bfd19cbede6 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:04.671202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:04.672359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.672845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.673355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.684534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:04.686248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:04.693973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:04.697809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:04.702110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:04.703058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:04.703948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:04.703948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:04.703948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:04.703948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 439/877] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:04.709135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:04.744061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:04.823253 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962bfd17-eaab-4648-8d53-2b3fe11c22da None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:04.833223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fc7b39-2a19-43b8-907b-77b3819d47ca None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:04.868702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbef1216-53a6-4628-b034-dfad37dcbae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:04.869569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbef1216-53a6-4628-b034-dfad37dcbae6 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.870017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbef1216-53a6-4628-b034-dfad37dcbae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.870395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fa52b3-e1fb-47a9-845d-b919dff46db0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:04.870594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbef1216-53a6-4628-b034-dfad37dcbae6 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.870997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fa52b3-e1fb-47a9-845d-b919dff46db0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.871273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fa52b3-e1fb-47a9-845d-b919dff46db0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.871544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fa52b3-e1fb-47a9-845d-b919dff46db0 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.871845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbef1216-53a6-4628-b034-dfad37dcbae6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:04.872179 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fa52b3-e1fb-47a9-845d-b919dff46db0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:04.872480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fa52b3-e1fb-47a9-845d-b919dff46db0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:04.873688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbef1216-53a6-4628-b034-dfad37dcbae6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:04.873863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fa52b3-e1fb-47a9-845d-b919dff46db0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:04.885695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 440/878] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:04.886900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd752dfc-63fb-4b78-9e81-e9a7815cd91c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.887181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd752dfc-63fb-4b78-9e81-e9a7815cd91c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.894498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd752dfc-63fb-4b78-9e81-e9a7815cd91c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.894498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 441/879] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:04.894498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd752dfc-63fb-4b78-9e81-e9a7815cd91c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:04.957648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 439/880] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:54:04 2022] POST /identity/v3/services => generated 251 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:04.957648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c6abf4-183a-42f9-b124-7633d9779d69 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:04.957648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c6abf4-183a-42f9-b124-7633d9779d69 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:04.957648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c6abf4-183a-42f9-b124-7633d9779d69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:04.971108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c8c6abf4-183a-42f9-b124-7633d9779d69 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:04.972416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c8c6abf4-183a-42f9-b124-7633d9779d69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:04.996738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:05.041838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd752dfc-63fb-4b78-9e81-e9a7815cd91c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:05.042767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:05.051942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 440/881] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:04 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:05.083898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:05.084554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:05.084832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:05.085109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:05.094069 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:05.095534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:05.109081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:05.110279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:05.111421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 442/882] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:54:04 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:05.116120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c6abf4-183a-42f9-b124-7633d9779d69 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:05.225130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c6abf4-183a-42f9-b124-7633d9779d69 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:05.261289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:05.261848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:05.262126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:05.262398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:05.263142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:05.263447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:05.264875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:05.282500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 441/883] 10.222.0.16 () {56 vars in 1149 bytes} [Thu Aug 25 04:54:05 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:05.287997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:05.344176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:05.400255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2a1926-544f-4a80-8cae-f5f64aa569c7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:05.407812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:05.446851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1eb9926-32e7-4231-a100-bff00fb07c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:05.447778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eb9926-32e7-4231-a100-bff00fb07c64 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:05.448388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eb9926-32e7-4231-a100-bff00fb07c64 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:05.448842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1eb9926-32e7-4231-a100-bff00fb07c64 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:05.460942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:05.469903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2366e880-7d05-460a-8e8b-51605702db61 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:05.508002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fdfcb2-0f4a-4783-8039-09ad0b79d714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:05.508636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fdfcb2-0f4a-4783-8039-09ad0b79d714 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:05.508936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fdfcb2-0f4a-4783-8039-09ad0b79d714 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:05.509320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fdfcb2-0f4a-4783-8039-09ad0b79d714 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:05.510096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fdfcb2-0f4a-4783-8039-09ad0b79d714 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:05.511637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fdfcb2-0f4a-4783-8039-09ad0b79d714 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:05.537353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eb9926-32e7-4231-a100-bff00fb07c64 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:05.539212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1eb9926-32e7-4231-a100-bff00fb07c64 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:05.612688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 442/884] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:54:05 2022] POST /identity/v3/services => generated 269 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:05.620029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ed3c39-6629-40f2-8280-2a5d7a4efb27 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:05.620362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ed3c39-6629-40f2-8280-2a5d7a4efb27 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:05.620641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ed3c39-6629-40f2-8280-2a5d7a4efb27 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:05.623468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 443/885] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:05 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:05.628765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdff104-2b75-419b-bccc-b04a02aafd8f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:05.629068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdff104-2b75-419b-bccc-b04a02aafd8f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:05.629372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdff104-2b75-419b-bccc-b04a02aafd8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:05.649988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7cdff104-2b75-419b-bccc-b04a02aafd8f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:05.651202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7cdff104-2b75-419b-bccc-b04a02aafd8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:05.661820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 443/886] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:05 2022] GET /identity/v3/auth/tokens => generated 1262 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:54:05.664968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1eb9926-32e7-4231-a100-bff00fb07c64 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:05.722111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:05.742512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 444/887] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:05 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:05.790626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1eb9926-32e7-4231-a100-bff00fb07c64 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:05.827502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:05.828131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:05.828406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:05.828672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:05.841056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:05.842549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:05.859330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:05.860317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:05.861235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:05.861235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:05.861235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:05.861235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 444/888] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:05.868202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdff104-2b75-419b-bccc-b04a02aafd8f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:05.909153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:05.969647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdff104-2b75-419b-bccc-b04a02aafd8f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:06.005628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:06.006637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.007620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.008103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.008954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:06.009452 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:06.011040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:06.016090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:06.024155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 445/889] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:06.029917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:06.131940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:06.145018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4969bc84-30a7-4196-9e11-0e8f2d698628 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:06.150549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:06.154664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fa0c03-ce21-4473-904b-4b97ed7d53de None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:06.192548 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:06.193630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.193907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.194187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.195234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:06.197103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:06.200477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:06.201661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.202144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.202628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:06.207460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:06.208440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:06.209304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 445/890] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:06 2022] GET /identity/v3/services/object-store => generated 93 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:06.213083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:06.215233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:06.216245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:06.240964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:06.242020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:06.243108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 446/891] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:06.247861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:06.327153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c30684-af1a-487c-9459-e98c857171fe None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:06.362889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:06.363806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.364292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.364727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.365510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:06.365838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:06.367744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:06.374440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8ad19e-d81a-4076-8871-4b621b3998c1 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:06.375533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 446/892] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:06 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:06.383261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:06.409610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:06.417383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:06.418310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.418617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.419031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.425000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:06.425000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:06.425000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:06.436228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 447/893] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:54:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:06.441666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:06.463407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:06.520839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e25bb-f4db-41da-9ef8-b2b603568192 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:06.558586 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1ba249-62fe-410a-95cc-c2a177fbfd59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:06.559205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1ba249-62fe-410a-95cc-c2a177fbfd59 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.559511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1ba249-62fe-410a-95cc-c2a177fbfd59 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.559763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1ba249-62fe-410a-95cc-c2a177fbfd59 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.560390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1ba249-62fe-410a-95cc-c2a177fbfd59 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:06.560755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1ba249-62fe-410a-95cc-c2a177fbfd59 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:06.562534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1ba249-62fe-410a-95cc-c2a177fbfd59 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:06.566318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8a0a72-381d-4489-855c-6414d5f78c63 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:06.567410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 447/894] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:06 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:06.610712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f454959-c389-4771-94ad-c418747c8c58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:06.611338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f454959-c389-4771-94ad-c418747c8c58 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.611616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f454959-c389-4771-94ad-c418747c8c58 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.611889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f454959-c389-4771-94ad-c418747c8c58 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.612641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f454959-c389-4771-94ad-c418747c8c58 None admin] RBAC: Adding query filter params (role.id=55d39bc11a0048728a909c34d6389fc1, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=b7ba48fe350b4644a9f5edd0729a0536) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:06.612951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f454959-c389-4771-94ad-c418747c8c58 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:06.614539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f454959-c389-4771-94ad-c418747c8c58 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:06.622351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 448/895] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:06 2022] GET /identity/v3/role_assignments?user.id=b7ba48fe350b4644a9f5edd0729a0536&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=55d39bc11a0048728a909c34d6389fc1 => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:06.943085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab4bf1-d1be-4778-8b3f-661fd0632909 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.943703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab4bf1-d1be-4778-8b3f-661fd0632909 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.943703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab4bf1-d1be-4778-8b3f-661fd0632909 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.945218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 448/896] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:06.950553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184ed00a-232a-4779-9325-22f86df07bbc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:06.950843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184ed00a-232a-4779-9325-22f86df07bbc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:06.951126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184ed00a-232a-4779-9325-22f86df07bbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:06.970242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-184ed00a-232a-4779-9325-22f86df07bbc None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:06.971435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-184ed00a-232a-4779-9325-22f86df07bbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:07.042747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 449/897] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:06 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:07.305793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ab4bf1-d1be-4778-8b3f-661fd0632909 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:07.395021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:07.415635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ab4bf1-d1be-4778-8b3f-661fd0632909 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:07.451921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:07.452565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:07.452848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:07.453121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:07.453726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:07.455111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:07.464574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:07.465575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:07.466429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 449/898] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:07 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:07.473679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184ed00a-232a-4779-9325-22f86df07bbc None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:07.488851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:07.578312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184ed00a-232a-4779-9325-22f86df07bbc None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:07.615073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68860e21-a0b2-478a-a366-8509f5cc0efe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:07.615682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68860e21-a0b2-478a-a366-8509f5cc0efe None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:07.615962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68860e21-a0b2-478a-a366-8509f5cc0efe None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:07.616233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68860e21-a0b2-478a-a366-8509f5cc0efe None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:07.616948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68860e21-a0b2-478a-a366-8509f5cc0efe None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:07.617251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68860e21-a0b2-478a-a366-8509f5cc0efe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:07.618726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68860e21-a0b2-478a-a366-8509f5cc0efe None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:07.623905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 450/899] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:07 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:07.629399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:07.737328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e560b817-6ead-47b4-a4d9-7563c56efcc4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:07.748750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74485f54-abd6-4ee0-8fa7-8d673cca6421 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:07.749109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74485f54-abd6-4ee0-8fa7-8d673cca6421 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:07.749398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74485f54-abd6-4ee0-8fa7-8d673cca6421 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:07.751200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 451/900] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:54:07.756739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352d43d6-2b14-4d76-b70c-cdab57c10edb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:07.757045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352d43d6-2b14-4d76-b70c-cdab57c10edb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:07.757394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352d43d6-2b14-4d76-b70c-cdab57c10edb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:07.778097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:07.778441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-352d43d6-2b14-4d76-b70c-cdab57c10edb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:07.779035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:07.779491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:07.779769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-352d43d6-2b14-4d76-b70c-cdab57c10edb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:07.779874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:07.780472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:07.780782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:07.782235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:07.787610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 450/901] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:07 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:07.794496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:07.813764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:07.881610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 452/902] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:07 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:07.924090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d8a9dd-273c-4ce8-a585-0826e813af6e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:07.959561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:07.960687 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:07.961026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:07.961298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:07.962266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] RBAC: Adding query filter params (interface=public, service_id=942bbf718df846a7a75791f1dedca4e2, region_id=RegionOne) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:07.962618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:07.963968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:07.969398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 451/903] 10.222.0.16 () {56 vars in 1251 bytes} [Thu Aug 25 04:54:07 2022] GET /identity/v3/endpoints?service_id=942bbf718df846a7a75791f1dedca4e2&interface=public®ion_id=RegionOne => generated 194 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:07.974915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352d43d6-2b14-4d76-b70c-cdab57c10edb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:07.983847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:08.079894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352d43d6-2b14-4d76-b70c-cdab57c10edb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:08.115082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8940497-e90e-48f9-80c1-b830d397bf47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:08.115752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8940497-e90e-48f9-80c1-b830d397bf47 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.116028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8940497-e90e-48f9-80c1-b830d397bf47 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.116297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8940497-e90e-48f9-80c1-b830d397bf47 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.116957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8940497-e90e-48f9-80c1-b830d397bf47 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:08.118346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8940497-e90e-48f9-80c1-b830d397bf47 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:08.124427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 453/904] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:54:07 2022] GET /identity/v3/services => generated 1268 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 04:54:08.149653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:08.247808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:08.275748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562c8bf6-0481-4e66-af57-f547aae9ad69 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:08.312811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:08.313449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.313754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.314024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.314605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:08.315954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:08.325293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:08.326315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:08.327211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 452/905] 10.222.0.16 () {56 vars in 1132 bytes} [Thu Aug 25 04:54:08 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:08.334421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8940497-e90e-48f9-80c1-b830d397bf47 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:08.410454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:08.441981 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8940497-e90e-48f9-80c1-b830d397bf47 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:08.476989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f004318-c091-4aad-8d72-ab4b3ddc7461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:08.477617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f004318-c091-4aad-8d72-ab4b3ddc7461 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.477906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f004318-c091-4aad-8d72-ab4b3ddc7461 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.478178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f004318-c091-4aad-8d72-ab4b3ddc7461 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.478812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f004318-c091-4aad-8d72-ab4b3ddc7461 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:08.479107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f004318-c091-4aad-8d72-ab4b3ddc7461 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:08.480462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f004318-c091-4aad-8d72-ab4b3ddc7461 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:08.485751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 454/906] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:08 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:08.491945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:08.533772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9611fc6-77e1-4b48-b62b-a55c4abec9d8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.534117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9611fc6-77e1-4b48-b62b-a55c4abec9d8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.534319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9611fc6-77e1-4b48-b62b-a55c4abec9d8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.535911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 455/907] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:08.540987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b02d6c-21d7-49d1-99e7-7a5d534abdb6 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.541252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b02d6c-21d7-49d1-99e7-7a5d534abdb6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.541522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b02d6c-21d7-49d1-99e7-7a5d534abdb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.559958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e7b02d6c-21d7-49d1-99e7-7a5d534abdb6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:08.561123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e7b02d6c-21d7-49d1-99e7-7a5d534abdb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:08.603446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da0efcc-b336-4e95-8c80-4d4883a55f65 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:08.630661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 456/908] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:08 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:08.634187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf7e5b-cc36-4c82-98eb-5b952031bb15 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.635026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf7e5b-cc36-4c82-98eb-5b952031bb15 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.635617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf7e5b-cc36-4c82-98eb-5b952031bb15 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.643823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 457/909] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:08.645425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628eb5a4-9895-47e3-a0eb-1fdf571ddcd8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.645738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628eb5a4-9895-47e3-a0eb-1fdf571ddcd8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.646046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628eb5a4-9895-47e3-a0eb-1fdf571ddcd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.655714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:08.656994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.657526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.658180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.659180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:08.659714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:08.661533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:08.665735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-628eb5a4-9895-47e3-a0eb-1fdf571ddcd8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:08.666959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-628eb5a4-9895-47e3-a0eb-1fdf571ddcd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:08.668452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 453/910] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:08 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:08.675656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:08.678758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:08.738869 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:08.747848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 458/911] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:08 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:08.792857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fbfe59-2190-4722-b05b-7cb7a15f2b80 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:08.827636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:08.828246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.828581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.828858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.829555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] RBAC: Adding query filter params (interface=public, service_id=52dfac8ec4d748279e43320a102b9382, region_id=RegionOne) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:08.829888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:08.831314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:08.838449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 454/912] 10.222.0.16 () {56 vars in 1251 bytes} [Thu Aug 25 04:54:08 2022] GET /identity/v3/endpoints?service_id=52dfac8ec4d748279e43320a102b9382&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:08.843507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628eb5a4-9895-47e3-a0eb-1fdf571ddcd8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:08.906351 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:08.954375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628eb5a4-9895-47e3-a0eb-1fdf571ddcd8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:08.974546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:08.990036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a76a78-9d40-4e02-ab06-703b8ee27d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:08.993510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a76a78-9d40-4e02-ab06-703b8ee27d95 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:08.993510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a76a78-9d40-4e02-ab06-703b8ee27d95 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:08.993510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a76a78-9d40-4e02-ab06-703b8ee27d95 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:08.993510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a76a78-9d40-4e02-ab06-703b8ee27d95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:08.993510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a76a78-9d40-4e02-ab06-703b8ee27d95 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:08.999496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 459/913] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:54:08 2022] GET /identity/v3/services => generated 1268 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 04:54:09.013770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f4f890-8faa-4333-8e8c-3ab7cd413f91 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:09.055924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcdcb86-fe67-4a74-9bb0-4dee22a0c6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:09.056870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcdcb86-fe67-4a74-9bb0-4dee22a0c6c8 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:09.057013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a76a78-9d40-4e02-ab06-703b8ee27d95 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:09.057201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcdcb86-fe67-4a74-9bb0-4dee22a0c6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:09.057514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcdcb86-fe67-4a74-9bb0-4dee22a0c6c8 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:09.058274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcdcb86-fe67-4a74-9bb0-4dee22a0c6c8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:09.059656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcdcb86-fe67-4a74-9bb0-4dee22a0c6c8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:09.129089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:09.135745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 455/914] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:54:08 2022] POST /identity/v3/services => generated 241 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:09.184175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a76a78-9d40-4e02-ab06-703b8ee27d95 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:09.234318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d680f00c-c01f-4a77-89ce-8732723a4f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:09.234967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d680f00c-c01f-4a77-89ce-8732723a4f07 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:09.235240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d680f00c-c01f-4a77-89ce-8732723a4f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:09.235506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d680f00c-c01f-4a77-89ce-8732723a4f07 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:09.248914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d680f00c-c01f-4a77-89ce-8732723a4f07 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-d680f00c-c01f-4a77-89ce-8732723a4f07 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:09.254432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:09.255508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:09.256448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 460/915] 10.222.0.16 () {56 vars in 1120 bytes} [Thu Aug 25 04:54:09 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:09.261999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcdcb86-fe67-4a74-9bb0-4dee22a0c6c8 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:09.376959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcdcb86-fe67-4a74-9bb0-4dee22a0c6c8 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:09.404297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:09.414185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:09.416859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:09.416859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:09.416859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:09.416859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:09.416859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:09.417771 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:09.427012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 456/916] 10.222.0.16 () {56 vars in 1115 bytes} [Thu Aug 25 04:54:09 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:09.432356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d680f00c-c01f-4a77-89ce-8732723a4f07 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:09.541000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d680f00c-c01f-4a77-89ce-8732723a4f07 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:09.556127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:09.576827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:09.577460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:09.581715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:09.581715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:09.589353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:09.590838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:09.609689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:09.610715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:09.612214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:09.612214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:09.612214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:09.612214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 461/917] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:09.617033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:09.718384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a09b0eb-d00a-483f-986b-2f18d0b4d783 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:09.754664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:09.755632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:09.756139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:09.756618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:09.758022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:09.758521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:09.760084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:09.771958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 457/918] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:09.777481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:09.886978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0456c144-ee0b-4b69-8a66-e4fbbd9afd2a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:09.922569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:09.923210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:09.923493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:09.923771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:09.931665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:09.933128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:09.944570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:09.946781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:09.948969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 462/919] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:54:09 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:09.953610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:10.001700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:10.059123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a845bf-95e5-4c3b-b01e-b644c343b0df None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:10.098418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b0014-3fd2-4abf-916f-c533ef582692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:10.099043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b0014-3fd2-4abf-916f-c533ef582692 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.099321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b0014-3fd2-4abf-916f-c533ef582692 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.099596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b0014-3fd2-4abf-916f-c533ef582692 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.100332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185b0014-3fd2-4abf-916f-c533ef582692 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:10.100673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185b0014-3fd2-4abf-916f-c533ef582692 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:10.102248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185b0014-3fd2-4abf-916f-c533ef582692 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:10.121248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 458/920] 10.222.0.16 () {56 vars in 1149 bytes} [Thu Aug 25 04:54:09 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:10.126508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:10.155033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:10.235363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b8e476-5e3a-41bd-9a41-50c3c2f597fa None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:10.263365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2c9bf-f3f5-4ca0-bd8e-2701e971b3cb None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.263670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2c9bf-f3f5-4ca0-bd8e-2701e971b3cb None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.264203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf2c9bf-f3f5-4ca0-bd8e-2701e971b3cb None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.266586 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 459/921] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:54:10.270837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c639a48-cda9-4625-a866-2b637c2bb92b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.271121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c639a48-cda9-4625-a866-2b637c2bb92b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.271397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c639a48-cda9-4625-a866-2b637c2bb92b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.275938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:10.276824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.277368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.277863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.289143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:10.290891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:10.291001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5c639a48-cda9-4625-a866-2b637c2bb92b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:10.292338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5c639a48-cda9-4625-a866-2b637c2bb92b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:10.306878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:10.307859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:10.308750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:10.308750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:10.308750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:10.310798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 463/922] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:10.318833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:10.367050 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 460/923] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:10.418457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:10.441110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee2334-e299-41b7-9c37-10e4dd4539df None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:10.479003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:10.479661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.479946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.480222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.480918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:10.481229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:10.482609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:10.495178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 464/924] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:10.500783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c639a48-cda9-4625-a866-2b637c2bb92b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:10.543823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:10.606577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c639a48-cda9-4625-a866-2b637c2bb92b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:10.642768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:10.644002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:10.644647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.644929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.645203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.654843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:10.656769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:10.671787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:10.672739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:10.674356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 461/925] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:10.679725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:10.764930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62f5492-c310-418f-ae78-3615fb190ba8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:10.800971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:10.802451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.802925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.803403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.804712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:10.806311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:10.815377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:10.816403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 25 04:54:10.817255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:10.818042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eee567a-67e8-4ade-89b2-4d73489ad2ff None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:10.825227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 465/926] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:10 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:10.826703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:10.829929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:10.837999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:10.861302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:10.862038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.862390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.862731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.863501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:10.863875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:10.867029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:10.885622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 462/927] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:54:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:10.890866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:10.951131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ef1f55-cb5f-468a-97b7-be2edb467651 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:10.990157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:10.991120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:10.992151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:10.992745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:10.993712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:10.994234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:10.995921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:11.001663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 466/928] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:10 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:11.005838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e4fbf8-aed6-4682-bb43-0cb837c079a6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:11.007330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:11.047561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbba8f3-552a-4dc3-b050-201ea534736c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:11.048230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbba8f3-552a-4dc3-b050-201ea534736c None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.048514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbba8f3-552a-4dc3-b050-201ea534736c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.048793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbba8f3-552a-4dc3-b050-201ea534736c None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/b7ba48fe350b4644a9f5edd0729a0536/roles/55d39bc11a0048728a909c34d6389fc1` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.051033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbba8f3-552a-4dc3-b050-201ea534736c None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=b7ba48fe350b4644a9f5edd0729a0536, role_id=55d39bc11a0048728a909c34d6389fc1)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:11.052467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbba8f3-552a-4dc3-b050-201ea534736c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:11.083380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:11.117820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c25e7b-0f27-4370-9fe4-535c8d0924a8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:11.149956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a602b-a7d2-4a92-a8a1-e76aff9b20e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:11.150567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a602b-a7d2-4a92-a8a1-e76aff9b20e6 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.150849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a602b-a7d2-4a92-a8a1-e76aff9b20e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.151123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a602b-a7d2-4a92-a8a1-e76aff9b20e6 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.153484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a602b-a7d2-4a92-a8a1-e76aff9b20e6 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:11.153484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a602b-a7d2-4a92-a8a1-e76aff9b20e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:11.153484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a602b-a7d2-4a92-a8a1-e76aff9b20e6 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:11.158013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 467/929] 10.222.0.16 () {56 vars in 1114 bytes} [Thu Aug 25 04:54:11 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:11.159142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48735ce5-0e78-48c2-8342-5722bc5dcfe2 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.159431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48735ce5-0e78-48c2-8342-5722bc5dcfe2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.159707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48735ce5-0e78-48c2-8342-5722bc5dcfe2 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.161061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 468/930] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:11 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:11.164485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48735ce5-0e78-48c2-8342-5722bc5dcfe2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:11.187359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 463/931] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:54:10 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/b7ba48fe350b4644a9f5edd0729a0536/roles/55d39bc11a0048728a909c34d6389fc1 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:54:11.188755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632c336f-d79f-4b03-8420-5a4a95001df3 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.189051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632c336f-d79f-4b03-8420-5a4a95001df3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.189326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632c336f-d79f-4b03-8420-5a4a95001df3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.228624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-632c336f-d79f-4b03-8420-5a4a95001df3 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:11.230400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-632c336f-d79f-4b03-8420-5a4a95001df3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:11.249840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:11.297400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48735ce5-0e78-48c2-8342-5722bc5dcfe2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:11.314549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 464/932] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:11 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:11.321863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300b958-4988-4a7b-8e64-82729cec1b8f None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.321863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300b958-4988-4a7b-8e64-82729cec1b8f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.321863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4300b958-4988-4a7b-8e64-82729cec1b8f None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.321863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 465/933] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:11.323745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a454d8ea-40db-4e96-832b-7f5336b653c0 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.324138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a454d8ea-40db-4e96-832b-7f5336b653c0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.324304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a454d8ea-40db-4e96-832b-7f5336b653c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.343437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:11.343774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-a454d8ea-40db-4e96-832b-7f5336b653c0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:11.345029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a454d8ea-40db-4e96-832b-7f5336b653c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:11.353567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07ab6a-959e-4e31-91a2-5d338c9d1bb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:11.354571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07ab6a-959e-4e31-91a2-5d338c9d1bb2 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.355124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07ab6a-959e-4e31-91a2-5d338c9d1bb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.357143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07ab6a-959e-4e31-91a2-5d338c9d1bb2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.357143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d07ab6a-959e-4e31-91a2-5d338c9d1bb2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:11.358036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d07ab6a-959e-4e31-91a2-5d338c9d1bb2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:11.417454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 466/934] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:11 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:11.449756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 469/935] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:54:11 2022] POST /identity/v3/endpoints => generated 346 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:11.603418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a454d8ea-40db-4e96-832b-7f5336b653c0 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:11.683835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:11.696234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07ab6a-959e-4e31-91a2-5d338c9d1bb2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:11.706723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a454d8ea-40db-4e96-832b-7f5336b653c0 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:11.751546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:11.752294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.752579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.752847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.753427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:11.754947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:11.764495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:11.765446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:11.766385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 467/936] 10.222.0.16 () {56 vars in 1132 bytes} [Thu Aug 25 04:54:11 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:11.771698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:11.833096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07ab6a-959e-4e31-91a2-5d338c9d1bb2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:11.880542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeb6361-5c04-4fe2-8008-b524975fdec3 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:11.884079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:11.885103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.885552 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.886015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.886750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:11.888772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:11.900382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:11.901383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:11.902296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 470/937] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:11 2022] GET /identity/v3/services/object-store => generated 93 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:11.907837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:11.926035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:11.926645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:11.926923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:11.927194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:11.927818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:11.928120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:11.929532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:11.931856 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:11.941281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 468/938] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:11 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:11.945757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:11.975119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:12.027086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b1dd47-b749-448e-82a8-2727a37d9aea None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:12.057809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fa77be-d6fe-4154-8c7a-365e597b8ee4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:12.062822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:12.063702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:12.064192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:12.064664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:12.065491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:12.066044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:12.067611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:12.074043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 471/939] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:11 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:12.079161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:12.100179 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:12.100861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:12.101196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:12.101441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:12.102147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:12.102451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:12.104026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:12.113903 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 469/940] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:11 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:12.114813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:12.182741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:12.210750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a8123f-d5f2-443c-a550-9fd2c7736cd5 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:12.227453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:12.243885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f3bc49-167d-4d1c-aad0-53021fd175cb None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:12.256932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:12.258098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:12.259054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:12.259803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:12.261060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:12.261956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:12.263890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:12.271324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 472/941] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:12 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:12.277204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:12.284483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1049fcfc-9792-4755-ac84-0539a57b30d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:12.285366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1049fcfc-9792-4755-ac84-0539a57b30d6 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:12.285881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1049fcfc-9792-4755-ac84-0539a57b30d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:12.286774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1049fcfc-9792-4755-ac84-0539a57b30d6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:12.287643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1049fcfc-9792-4755-ac84-0539a57b30d6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:12.289695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1049fcfc-9792-4755-ac84-0539a57b30d6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:12.383131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6038dbd-6ccb-47cf-b2ad-6cef98b45769 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:12.410254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 470/942] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:54:12 2022] POST /identity/v3/endpoints => generated 347 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:54:12.433751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51796f8b-7bf6-4dd0-a8e1-a18aaa01f55e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:12.434294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51796f8b-7bf6-4dd0-a8e1-a18aaa01f55e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:12.434619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51796f8b-7bf6-4dd0-a8e1-a18aaa01f55e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:12.434890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51796f8b-7bf6-4dd0-a8e1-a18aaa01f55e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:12.435655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51796f8b-7bf6-4dd0-a8e1-a18aaa01f55e None admin] RBAC: Adding query filter params (interface=public, service_id=32f2d0db30784e30b1df7575ad5baae3, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:12.436005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51796f8b-7bf6-4dd0-a8e1-a18aaa01f55e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:12.437383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51796f8b-7bf6-4dd0-a8e1-a18aaa01f55e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:12.444210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 473/943] 10.222.0.16 () {56 vars in 1251 bytes} [Thu Aug 25 04:54:12 2022] GET /identity/v3/endpoints?service_id=32f2d0db30784e30b1df7575ad5baae3&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:12.450324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1049fcfc-9792-4755-ac84-0539a57b30d6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:12.565952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1049fcfc-9792-4755-ac84-0539a57b30d6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:12.588728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:12.603388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0260b20-77fd-4fbc-9ee9-333050eee009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:12.608575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0260b20-77fd-4fbc-9ee9-333050eee009 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:12.608575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0260b20-77fd-4fbc-9ee9-333050eee009 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:12.608575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0260b20-77fd-4fbc-9ee9-333050eee009 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:12.608575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0260b20-77fd-4fbc-9ee9-333050eee009 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:12.608575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0260b20-77fd-4fbc-9ee9-333050eee009 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:12.613370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 471/944] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:54:12 2022] GET /identity/v3/services => generated 1497 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 04:54:13.164867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae2d3ec-3c7a-42b8-b577-4ceaea54b471 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:13.165411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae2d3ec-3c7a-42b8-b577-4ceaea54b471 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:13.165411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae2d3ec-3c7a-42b8-b577-4ceaea54b471 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:13.166975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 474/945] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:13.172271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d001686-4bf2-4e77-ad95-2d7b4995e1a1 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:13.172672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d001686-4bf2-4e77-ad95-2d7b4995e1a1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:13.172849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d001686-4bf2-4e77-ad95-2d7b4995e1a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:13.191455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5d001686-4bf2-4e77-ad95-2d7b4995e1a1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:13.192622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5d001686-4bf2-4e77-ad95-2d7b4995e1a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:13.266306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 472/946] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:13 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:13.464079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2b086e-d19d-42c8-b0b3-a8c07cee8e1a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:13.464312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2b086e-d19d-42c8-b0b3-a8c07cee8e1a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:13.464579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2b086e-d19d-42c8-b0b3-a8c07cee8e1a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:13.466288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 475/947] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:13.471518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2203ee3e-07f7-4bbe-8f15-4960629bfe6e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:13.471810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2203ee3e-07f7-4bbe-8f15-4960629bfe6e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:13.472083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2203ee3e-07f7-4bbe-8f15-4960629bfe6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:13.491598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2203ee3e-07f7-4bbe-8f15-4960629bfe6e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:13.492776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2203ee3e-07f7-4bbe-8f15-4960629bfe6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:13.544207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2b086e-d19d-42c8-b0b3-a8c07cee8e1a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:13.570693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 473/948] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:13 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:13.595444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:13.660715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2b086e-d19d-42c8-b0b3-a8c07cee8e1a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:13.696857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1426043e-2a37-49b7-88e8-069f19266aee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:13.697846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426043e-2a37-49b7-88e8-069f19266aee None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:13.698409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426043e-2a37-49b7-88e8-069f19266aee None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:13.698887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1426043e-2a37-49b7-88e8-069f19266aee None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:13.709695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1426043e-2a37-49b7-88e8-069f19266aee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1426043e-2a37-49b7-88e8-069f19266aee None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:13.715646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:13.716570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:13.717431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:13.720102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 476/949] 10.222.0.16 () {56 vars in 1120 bytes} [Thu Aug 25 04:54:13 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:13.727168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2203ee3e-07f7-4bbe-8f15-4960629bfe6e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:13.776841 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:13.832924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2203ee3e-07f7-4bbe-8f15-4960629bfe6e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:13.868574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:13.869191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:13.869474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:13.869774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:13.870517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:13.870883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:13.872301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:13.880821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 474/950] 10.222.0.16 () {56 vars in 1115 bytes} [Thu Aug 25 04:54:13 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:13.883385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1426043e-2a37-49b7-88e8-069f19266aee None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:13.886199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:14.010704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:14.015979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1426043e-2a37-49b7-88e8-069f19266aee None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:14.019534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a87aa8-4ffb-4649-99b1-e5756a6a42b2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:14.028678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:14.058880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:14.059584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.059926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.060261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.061093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:14.062200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:14.062742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:14.063237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.063706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.064159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.065563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING keystone.server.flask.common [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 25 04:54:14.076742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:14.078501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:14.094061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:14.099741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:14.101226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:14.101226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:14.101226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:14.101226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 475/951] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:14.102073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:14.119532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.sql.core [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 25 04:54:14.119532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 25 04:54:14.119532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [parameters: {'id': 'f538e6abfa1b4669ad8be21eaf5e5585', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 25 04:54:14.119532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81762) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f538e6abfa1b4669ad8be21eaf5e5585.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f538e6abfa1b4669ad8be21eaf5e5585. Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result = self._query(query) Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application conn.query(q) Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 25 04:54:14.130938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result.read() Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 25 04:54:14.131812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application next(self.gen) Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application yield resource Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application next(self.gen) Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.session.rollback() Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.force_reraise() Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise self.value Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application session.commit() Aug 25 04:54:14.132889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._prepare_impl() Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.session.flush() Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._flush(objects) Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application compat.raise_( Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application flush_context.execute() Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 25 04:54:14.139769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rec.execute(self) Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 25 04:54:14.140787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result = self._query(query) Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application conn.query(q) Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result.read() Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 25 04:54:14.142901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [parameters: {'id': 'f538e6abfa1b4669ad8be21eaf5e5585', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:14.145063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:14.147172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f538e6abfa1b4669ad8be21eaf5e5585. Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:14.149334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 477/952] 10.222.0.16 () {60 vars in 1140 bytes} [Thu Aug 25 04:54:13 2022] POST /identity/v3/projects => generated 262 bytes in 253 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:14.152353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:14.222192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1021c1-5e5b-4eb2-9c9b-b355f1a7c282 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:14.259452 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edcaade-fc3c-4009-ad52-74e24ddb61d0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:14.261398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:14.262117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.262497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.262886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.263856 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:14.264029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:14.265558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:14.283418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 476/953] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:14.284895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:14.298402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:14.299260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.299786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.300208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.306568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:14.308615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:14.310442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:14.322246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:14.323179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:14.324724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:14.324724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:14.324724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:14.324724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:14.324724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:14.324724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:14.324724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:14.324724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:14.331983 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 478/954] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:14.333075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:14.352381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:14.398963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88aeccab-728b-4a00-8e2d-3cbe72a57a57 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:14.434233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d123d4ea-7dec-4068-b662-d4e9214eefca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:14.434866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d123d4ea-7dec-4068-b662-d4e9214eefca None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.435140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d123d4ea-7dec-4068-b662-d4e9214eefca None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.435410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d123d4ea-7dec-4068-b662-d4e9214eefca None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.442711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d123d4ea-7dec-4068-b662-d4e9214eefca None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:14.444061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13f2396-5484-41bb-9667-574d7a388552 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:14.444600 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d123d4ea-7dec-4068-b662-d4e9214eefca None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-d123d4ea-7dec-4068-b662-d4e9214eefca None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:14.456134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:14.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 25 04:54:14.458527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:14.462756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 477/955] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:54:14 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:14.464018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b603ede9-adbd-4f1c-a80b-7b2304dd9158 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.464308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b603ede9-adbd-4f1c-a80b-7b2304dd9158 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.464579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b603ede9-adbd-4f1c-a80b-7b2304dd9158 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.470159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 478/956] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:14.470159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b603ede9-adbd-4f1c-a80b-7b2304dd9158 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:14.486007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2def3afa-0375-4997-8281-9675ecbc4f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:14.486635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2def3afa-0375-4997-8281-9675ecbc4f74 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.486939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2def3afa-0375-4997-8281-9675ecbc4f74 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.487188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2def3afa-0375-4997-8281-9675ecbc4f74 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.487857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2def3afa-0375-4997-8281-9675ecbc4f74 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:14.488159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2def3afa-0375-4997-8281-9675ecbc4f74 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:14.489584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2def3afa-0375-4997-8281-9675ecbc4f74 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:14.510124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 479/957] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:54:14 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:14.511548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c0f902-e85a-46f1-a468-45883a7f4881 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.511779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c0f902-e85a-46f1-a468-45883a7f4881 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.512055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c0f902-e85a-46f1-a468-45883a7f4881 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.531781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c9c0f902-e85a-46f1-a468-45883a7f4881 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:14.533426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c9c0f902-e85a-46f1-a468-45883a7f4881 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:14.572855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:14.611642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b603ede9-adbd-4f1c-a80b-7b2304dd9158 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:14.623487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 480/958] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:14.628145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da280bce-108a-4bef-9863-3e16d35d8b56 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.628430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da280bce-108a-4bef-9863-3e16d35d8b56 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.628698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da280bce-108a-4bef-9863-3e16d35d8b56 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.630356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 481/959] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:14.637195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858f1191-998d-41de-9155-9b97b1303b05 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.637692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858f1191-998d-41de-9155-9b97b1303b05 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.637999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858f1191-998d-41de-9155-9b97b1303b05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.648546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:14.662836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:14.664384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.664662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.664987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.665674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-858f1191-998d-41de-9155-9b97b1303b05 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:14.665801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:14.666131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:14.667408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-858f1191-998d-41de-9155-9b97b1303b05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:14.667725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:14.694770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 479/960] 10.222.0.16 () {56 vars in 1149 bytes} [Thu Aug 25 04:54:14 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:14.700411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:14.763648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 482/961] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:14.820091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e73c8c-1498-4254-8612-d40ac93824fa None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:14.855332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4f7916-f257-47e3-93c0-56f9dd28bcec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:14.856251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4f7916-f257-47e3-93c0-56f9dd28bcec None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:14.856787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4f7916-f257-47e3-93c0-56f9dd28bcec None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:14.857249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4f7916-f257-47e3-93c0-56f9dd28bcec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:14.860928 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:14.896514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858f1191-998d-41de-9155-9b97b1303b05 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:14.937669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4f7916-f257-47e3-93c0-56f9dd28bcec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:14.939708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4f7916-f257-47e3-93c0-56f9dd28bcec None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.003105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858f1191-998d-41de-9155-9b97b1303b05 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.039037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.039598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.039878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.040152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] PATH_INFO: `/v3/services/compute` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.040742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.042142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.044070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:15.052225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:15.056180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:15.057056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 483/962] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:14 2022] GET /identity/v3/services/compute => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:15.057920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.071253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 480/963] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:14 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 25 04:54:15.076666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4f7916-f257-47e3-93c0-56f9dd28bcec None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.112196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:15.168972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3835145a-ce37-42da-8a21-623b7832663e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.182354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4f7916-f257-47e3-93c0-56f9dd28bcec None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.205384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.206148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.206434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.206710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.207297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.208662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:15.220422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:15.221662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:15.222715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 484/964] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:15.223697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.232001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.232629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.233024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.233769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.234430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] RBAC: Adding query filter params (name=compute) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:15.234742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.236531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.242761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 481/965] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:15.243469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.294224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:15.339022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9762ed6-2b41-4bd6-ad6d-b45099ac8770 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.348237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:15.351781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9453b9b3-9680-4476-946c-79808b8fdbff None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.374217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.374916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.375190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.375468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.386654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.391113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.391236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.391236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.391384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.391467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.391467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:15.391467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.391467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.396089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 482/966] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:15.396701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:15.408360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:15.409335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:15.410285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:15.410285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:15.410285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:15.410285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 485/967] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:15.410528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.503543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef852f95-b834-48d1-94bc-a14ff9631dc0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.516264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013417fe-d184-45e4-a208-b49112165d92 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.530907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:15.537438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.538668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.538951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.539224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.544847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:15.544847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.544847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.553947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 483/968] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:15.553947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.561083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.561746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.562029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.562301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.562949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:15.563460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.565266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.577247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 486/969] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:15.578381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.591574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:15.688442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2656f2-f285-42f1-9284-bd7a64809fc4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.706497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e8066-7ba6-4c83-93d7-7d61b53760ab None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.729484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.730422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.730885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.731376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.732478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:15.732924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.734832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.744225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be99d0ba-9a10-44a2-bfc6-b396b24ef837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.745071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be99d0ba-9a10-44a2-bfc6-b396b24ef837 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.745490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be99d0ba-9a10-44a2-bfc6-b396b24ef837 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.746325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be99d0ba-9a10-44a2-bfc6-b396b24ef837 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.747424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be99d0ba-9a10-44a2-bfc6-b396b24ef837 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.748906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be99d0ba-9a10-44a2-bfc6-b396b24ef837 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.749303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 484/970] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:15.755640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.853305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:15.854403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 487/971] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:54:15 2022] POST /identity/v3/endpoints => generated 349 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:54:15.871705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff95643-c1a0-4777-be9c-6f34463f7cb0 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:15.919852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:15.920512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:15.920797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:15.921068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:15.929723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:15.931968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:15.945372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:15.946350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:15.947248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 485/972] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:15.952912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be99d0ba-9a10-44a2-bfc6-b396b24ef837 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:15.994232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:16.064966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be99d0ba-9a10-44a2-bfc6-b396b24ef837 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:16.101458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762ecdc8-4a29-4d5d-a4a5-458f7a57f272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:16.102097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762ecdc8-4a29-4d5d-a4a5-458f7a57f272 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:16.102382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762ecdc8-4a29-4d5d-a4a5-458f7a57f272 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:16.102660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762ecdc8-4a29-4d5d-a4a5-458f7a57f272 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:16.103417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762ecdc8-4a29-4d5d-a4a5-458f7a57f272 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:16.103670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762ecdc8-4a29-4d5d-a4a5-458f7a57f272 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:16.105107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762ecdc8-4a29-4d5d-a4a5-458f7a57f272 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:16.117940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 488/973] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:54:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:16.123833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:16.229821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a72b60c-14fc-4200-a5bf-b711b6c1f0cc None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:16.266708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2978463e-754e-4935-82d2-c13db5566dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:16.267326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978463e-754e-4935-82d2-c13db5566dae None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:16.267590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978463e-754e-4935-82d2-c13db5566dae None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:16.267861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978463e-754e-4935-82d2-c13db5566dae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:16.268588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2978463e-754e-4935-82d2-c13db5566dae None admin] RBAC: Adding query filter params (role.id=55d39bc11a0048728a909c34d6389fc1, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=b7ba48fe350b4644a9f5edd0729a0536) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:16.268896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2978463e-754e-4935-82d2-c13db5566dae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:16.270359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2978463e-754e-4935-82d2-c13db5566dae None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:16.276763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 486/974] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:16 2022] GET /identity/v3/role_assignments?user.id=b7ba48fe350b4644a9f5edd0729a0536&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=55d39bc11a0048728a909c34d6389fc1 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:16.632930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8b94ba-4283-41b5-8fac-64c4ee2c108b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:16.633491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8b94ba-4283-41b5-8fac-64c4ee2c108b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:16.633491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8b94ba-4283-41b5-8fac-64c4ee2c108b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:16.635155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 489/975] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:16.640845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17566094-1e8a-4c09-ad08-8935c110153c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:16.641134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17566094-1e8a-4c09-ad08-8935c110153c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:16.641405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17566094-1e8a-4c09-ad08-8935c110153c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:16.660234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-17566094-1e8a-4c09-ad08-8935c110153c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:16.661429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-17566094-1e8a-4c09-ad08-8935c110153c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:16.664143 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:16.734494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 487/976] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:16 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:17.003646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8b94ba-4283-41b5-8fac-64c4ee2c108b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:17.109003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:17.117451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8b94ba-4283-41b5-8fac-64c4ee2c108b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:17.151555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:17.152166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:17.152450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:17.152722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:17.153369 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:17.154826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:17.204550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.sql.core [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 25 04:54:17.204550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 25 04:54:17.204550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [parameters: {'user_id': 'ff39b63651ce4a3dad984513ff455ce7', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 25 04:54:17.204550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81762) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result = self._query(query) Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application conn.query(q) Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result.read() Aug 25 04:54:17.212035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 25 04:54:17.213067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application next(self.gen) Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application yield resource Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application next(self.gen) Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.session.rollback() Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.force_reraise() Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise self.value Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application session.commit() Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 25 04:54:17.214055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._prepare_impl() Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.session.flush() Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._flush(objects) Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application compat.raise_( Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application flush_context.execute() Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rec.execute(self) Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 25 04:54:17.214955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 25 04:54:17.215851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result = self._query(query) Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application conn.query(q) Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application result.read() Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 25 04:54:17.216774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [parameters: {'user_id': 'ff39b63651ce4a3dad984513ff455ce7', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:17.218686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 25 04:54:17.220554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 25 04:54:17.234775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 25 04:54:17.234775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 25 04:54:17.234775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 25 04:54:17.234775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 25 04:54:17.234775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 25 04:54:17.234775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:17.234775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 490/977] 10.222.0.16 () {60 vars in 1131 bytes} [Thu Aug 25 04:54:17 2022] POST /identity/v3/users => generated 160 bytes in 210 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:17.235350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17566094-1e8a-4c09-ad08-8935c110153c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:17.327992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17566094-1e8a-4c09-ad08-8935c110153c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:17.365522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e916e8-458c-43f8-852f-2487edf6420a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:17.366302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e916e8-458c-43f8-852f-2487edf6420a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:17.366571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e916e8-458c-43f8-852f-2487edf6420a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:17.366845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e916e8-458c-43f8-852f-2487edf6420a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:17.374546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e916e8-458c-43f8-852f-2487edf6420a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:17.376017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e916e8-458c-43f8-852f-2487edf6420a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:17.385477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-51e916e8-458c-43f8-852f-2487edf6420a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:17.388245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:17.390312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:17.392302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 488/978] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:54:17 2022] GET /identity/v3/users/cinder => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:17.395601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:17.507373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3c8b7a-117a-44af-a95d-39c88b960f31 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:17.542973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264b0acc-dd7a-4417-a831-07153fc7337e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:17.543584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b0acc-dd7a-4417-a831-07153fc7337e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:17.543863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b0acc-dd7a-4417-a831-07153fc7337e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:17.544134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b0acc-dd7a-4417-a831-07153fc7337e None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:17.544841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264b0acc-dd7a-4417-a831-07153fc7337e None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:17.545200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264b0acc-dd7a-4417-a831-07153fc7337e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:17.546873 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264b0acc-dd7a-4417-a831-07153fc7337e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:17.575827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa46f31-ab8f-410c-8523-74fde547ddc2 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:17.575827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa46f31-ab8f-410c-8523-74fde547ddc2 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:17.575827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa46f31-ab8f-410c-8523-74fde547ddc2 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:17.576658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 489/979] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:17.581700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5ee8ae-3c48-4023-922c-03e87dbd7e99 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:17.582221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5ee8ae-3c48-4023-922c-03e87dbd7e99 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:17.582698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5ee8ae-3c48-4023-922c-03e87dbd7e99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:17.587910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 491/980] 10.222.0.16 () {56 vars in 1101 bytes} [Thu Aug 25 04:54:17 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:17.602361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2d5ee8ae-3c48-4023-922c-03e87dbd7e99 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:17.603991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2d5ee8ae-3c48-4023-922c-03e87dbd7e99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:17.680471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 490/981] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:17.935567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011a8d7-d8fb-4c5c-9002-5a005de52063 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:17.936111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011a8d7-d8fb-4c5c-9002-5a005de52063 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:17.936111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011a8d7-d8fb-4c5c-9002-5a005de52063 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:17.937515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 492/982] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:17.942660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc6b30-3020-4920-81d7-48c7b7188a6d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:17.942955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc6b30-3020-4920-81d7-48c7b7188a6d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:17.943232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc6b30-3020-4920-81d7-48c7b7188a6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:17.948386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a011a8d7-d8fb-4c5c-9002-5a005de52063 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:17.964680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6ffc6b30-3020-4920-81d7-48c7b7188a6d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:17.964857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:17.965573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6ffc6b30-3020-4920-81d7-48c7b7188a6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:18.040695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 491/983] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:18.054124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a011a8d7-d8fb-4c5c-9002-5a005de52063 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:18.089410 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3293b98-1817-4496-aa56-b811137b9cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:18.090062 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3293b98-1817-4496-aa56-b811137b9cf7 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.090344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3293b98-1817-4496-aa56-b811137b9cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.090617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3293b98-1817-4496-aa56-b811137b9cf7 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.091269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3293b98-1817-4496-aa56-b811137b9cf7 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:18.092620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3293b98-1817-4496-aa56-b811137b9cf7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:18.166287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 493/984] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:54:17 2022] POST /identity/v3/services => generated 242 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:18.306329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fc7b4-0f45-44c8-9e61-ce62c787667b None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.306644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fc7b4-0f45-44c8-9e61-ce62c787667b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.306895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7fc7b4-0f45-44c8-9e61-ce62c787667b None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.308245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 492/985] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:18.308468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3293b98-1817-4496-aa56-b811137b9cf7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:18.313460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1a1f14-4cef-47b4-9fe0-802020336a56 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.313770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1a1f14-4cef-47b4-9fe0-802020336a56 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.314047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1a1f14-4cef-47b4-9fe0-802020336a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.332520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8c1a1f14-4cef-47b4-9fe0-802020336a56 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:18.333762 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8c1a1f14-4cef-47b4-9fe0-802020336a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:18.389444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:18.390432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:18.415170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 493/986] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:18 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:18.420343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3293b98-1817-4496-aa56-b811137b9cf7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:18.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:18.454937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.455691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.455691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.456667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:18.458346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:18.467507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:18.470596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:18.471489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 494/987] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:18 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:18.474513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1a1f14-4cef-47b4-9fe0-802020336a56 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:18.573855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1a1f14-4cef-47b4-9fe0-802020336a56 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:18.608026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:18.608635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.608914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.609187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.609833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:18.610137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:18.611500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:18.616529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 494/988] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:18 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:18.621994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:18.681409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:18.729961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c73eb53-5819-4bed-9c71-896347d9be1a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:18.737831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:18.766244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:18.767065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.767192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.767383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.768001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:18.768435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:18.769676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:18.774908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 495/989] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:18 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:18.780523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:18.783305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:18.792294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da5e5d7-43d6-4c63-872f-52fcb0aea97f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:18.830276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:18.831041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.831360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.831590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] PATH_INFO: `/v3/services/image` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.832184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:18.833543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:18.842795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:18.845902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:18.846974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 495/990] 10.222.0.16 () {56 vars in 1105 bytes} [Thu Aug 25 04:54:18 2022] GET /identity/v3/services/image => generated 86 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:18.849763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:18.891777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235a41a0-4f1e-491b-a24d-a5c336b5c248 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:18.924708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:18.925310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.925619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.925888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.926582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] RBAC: Adding query filter params (interface=admin, service_id=32f2d0db30784e30b1df7575ad5baae3, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:18.926886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:18.928229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:18.933696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 496/991] 10.222.0.16 () {56 vars in 1249 bytes} [Thu Aug 25 04:54:18 2022] GET /identity/v3/endpoints?service_id=32f2d0db30784e30b1df7575ad5baae3&interface=admin®ion_id=RegionOne => generated 193 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:18.939093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:18.958556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306488b6-1944-4ce4-b954-f65cafb9f196 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:18.973005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:18.994085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:18.994629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:18.994908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:18.995181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:18.995803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] RBAC: Adding query filter params (name=image) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:18.996107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:18.997524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:19.002496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 496/992] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:18 2022] GET /identity/v3/services?name=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:19.005668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:19.008582 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:19.047758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f39e3d-e19c-4ddf-92fe-2364d8929348 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:19.088088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4684e45-ddc2-47bf-91f7-405974bf1198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:19.089352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4684e45-ddc2-47bf-91f7-405974bf1198 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:19.089897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4684e45-ddc2-47bf-91f7-405974bf1198 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:19.090397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4684e45-ddc2-47bf-91f7-405974bf1198 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:19.091192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4684e45-ddc2-47bf-91f7-405974bf1198 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:19.092648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4684e45-ddc2-47bf-91f7-405974bf1198 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:19.100044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 497/993] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:54:18 2022] GET /identity/v3/services => generated 1727 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 04:54:19.114543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668b9d14-05a2-4c49-ac48-89b7abda7f1f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:19.154364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec432a7-d489-4386-91e4-7924527ad5b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:19.155006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec432a7-d489-4386-91e4-7924527ad5b5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:19.155288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec432a7-d489-4386-91e4-7924527ad5b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:19.155564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec432a7-d489-4386-91e4-7924527ad5b5 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:19.156208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec432a7-d489-4386-91e4-7924527ad5b5 None admin] RBAC: Adding query filter params (type=image) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:19.156515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec432a7-d489-4386-91e4-7924527ad5b5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:19.157949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec432a7-d489-4386-91e4-7924527ad5b5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:19.163961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 497/994] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:19 2022] GET /identity/v3/services?type=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:20.096430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ca918f-9042-4aff-b2b0-79098d5f9ee6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:20.096946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ca918f-9042-4aff-b2b0-79098d5f9ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:20.096946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ca918f-9042-4aff-b2b0-79098d5f9ee6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:20.098377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 498/995] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:20.104055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20dc6c6-7637-40db-bcb9-ca0b45315781 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:20.104463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20dc6c6-7637-40db-bcb9-ca0b45315781 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:20.104639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20dc6c6-7637-40db-bcb9-ca0b45315781 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:20.122626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e20dc6c6-7637-40db-bcb9-ca0b45315781 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:20.123986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e20dc6c6-7637-40db-bcb9-ca0b45315781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:20.161985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:20.197346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 498/996] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:20 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:20.464716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ca918f-9042-4aff-b2b0-79098d5f9ee6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:20.568379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ca918f-9042-4aff-b2b0-79098d5f9ee6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:20.597308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:20.603487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:20.604171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:20.604451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:20.604716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:20.605293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:20.606676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:20.615440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:20.616938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:20.618176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:20.622174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 499/997] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:20.623543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20dc6c6-7637-40db-bcb9-ca0b45315781 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:20.725829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20dc6c6-7637-40db-bcb9-ca0b45315781 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:20.762249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:20.762865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:20.763149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:20.763421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:20.764053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:20.764354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:20.765820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:20.775355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 499/998] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:54:20 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:20.776311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:20.879875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cf368c-7796-4146-a4c9-30770e8dd4d9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:20.913846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e020d-6a90-457c-82ad-201989c61ae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:20.914449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e020d-6a90-457c-82ad-201989c61ae3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:20.914724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e020d-6a90-457c-82ad-201989c61ae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:20.914990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e020d-6a90-457c-82ad-201989c61ae3 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:20.915678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e020d-6a90-457c-82ad-201989c61ae3 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:20.915982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e020d-6a90-457c-82ad-201989c61ae3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:20.917323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e020d-6a90-457c-82ad-201989c61ae3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:20.922235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 500/999] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:54:20 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:20.927858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:20.964236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:21.008123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d73812-8fd8-45bb-899a-0635e4809e37 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.008681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d73812-8fd8-45bb-899a-0635e4809e37 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.008681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d73812-8fd8-45bb-899a-0635e4809e37 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.010069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 501/1000] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:21.014961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64002312-872c-495c-bba5-3b68911df5ce None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.015301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64002312-872c-495c-bba5-3b68911df5ce None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.015581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64002312-872c-495c-bba5-3b68911df5ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.032756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ee807a-08cf-4092-99c6-c586927dc2b8 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:21.034260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-64002312-872c-495c-bba5-3b68911df5ce None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:21.035771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-64002312-872c-495c-bba5-3b68911df5ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:21.068628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87afc246-f5f4-4d9a-89e9-1270ff97511d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:21.069500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87afc246-f5f4-4d9a-89e9-1270ff97511d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.069808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87afc246-f5f4-4d9a-89e9-1270ff97511d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.070082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87afc246-f5f4-4d9a-89e9-1270ff97511d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.070799 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87afc246-f5f4-4d9a-89e9-1270ff97511d None admin] RBAC: Adding query filter params (interface=public, service_id=15b9ec92b03f4a3b85df06f509f68f07, region_id=RegionOne) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:21.071113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87afc246-f5f4-4d9a-89e9-1270ff97511d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:21.072478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87afc246-f5f4-4d9a-89e9-1270ff97511d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:21.078087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 500/1001] 10.222.0.16 () {56 vars in 1251 bytes} [Thu Aug 25 04:54:20 2022] GET /identity/v3/endpoints?service_id=15b9ec92b03f4a3b85df06f509f68f07&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:21.079136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1116d449-c330-4b96-a537-134f3980ec07 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.079477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1116d449-c330-4b96-a537-134f3980ec07 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.079759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1116d449-c330-4b96-a537-134f3980ec07 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.080954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 501/1002] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:21 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:21.083302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1116d449-c330-4b96-a537-134f3980ec07 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:21.096017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:21.109966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 502/1003] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:21 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:21.111271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4893f8-5ded-445c-a285-d93c1098bd3c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.111617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4893f8-5ded-445c-a285-d93c1098bd3c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.111954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4893f8-5ded-445c-a285-d93c1098bd3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.129893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cd4893f8-5ded-445c-a285-d93c1098bd3c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:21.131228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cd4893f8-5ded-445c-a285-d93c1098bd3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:21.187942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:21.194019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1116d449-c330-4b96-a537-134f3980ec07 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:21.210252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 503/1004] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:21 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:21.237369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e0efb9-79f4-4663-8ac7-7e79f69f23a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:21.238052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0efb9-79f4-4663-8ac7-7e79f69f23a9 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.238472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0efb9-79f4-4663-8ac7-7e79f69f23a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.238884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e0efb9-79f4-4663-8ac7-7e79f69f23a9 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.239547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e0efb9-79f4-4663-8ac7-7e79f69f23a9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:21.241009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e0efb9-79f4-4663-8ac7-7e79f69f23a9 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:21.247701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 502/1005] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:54:21 2022] GET /identity/v3/services => generated 1727 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 04:54:21.377870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4893f8-5ded-445c-a285-d93c1098bd3c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:21.482654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e0efb9-79f4-4663-8ac7-7e79f69f23a9 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:21.483631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4893f8-5ded-445c-a285-d93c1098bd3c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:21.503331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:21.522125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:21.523028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.523308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.523581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.524169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:21.526266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:21.534942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:21.536136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:21.537103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 504/1006] 10.222.0.16 () {56 vars in 1126 bytes} [Thu Aug 25 04:54:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:21.542345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:21.601720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e0efb9-79f4-4663-8ac7-7e79f69f23a9 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:21.637422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd2265-6f09-42f7-81ca-8081212f033b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:21.637982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd2265-6f09-42f7-81ca-8081212f033b None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.638264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd2265-6f09-42f7-81ca-8081212f033b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.638522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd2265-6f09-42f7-81ca-8081212f033b None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.639151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd2265-6f09-42f7-81ca-8081212f033b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:21.640525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd2265-6f09-42f7-81ca-8081212f033b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:21.649937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f01187b-6124-4479-87b5-de7ee5a6651c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:21.684923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:21.685553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.685894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.686220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.686849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:21.687149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:21.688486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:21.693621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 505/1007] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:21 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:21.699116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:21.745709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:21.757676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 503/1008] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:54:21 2022] POST /identity/v3/services => generated 242 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:21.804300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2951575-042c-4fc0-ae6f-c901413b8750 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:21.847928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dede2d59-9cab-4909-b57a-1415e73bdf78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:21.848548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede2d59-9cab-4909-b57a-1415e73bdf78 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:21.848825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede2d59-9cab-4909-b57a-1415e73bdf78 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:21.849091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dede2d59-9cab-4909-b57a-1415e73bdf78 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:21.849814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dede2d59-9cab-4909-b57a-1415e73bdf78 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:21.850659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dede2d59-9cab-4909-b57a-1415e73bdf78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:21.852038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dede2d59-9cab-4909-b57a-1415e73bdf78 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:21.857973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 506/1009] 10.222.0.16 () {56 vars in 1122 bytes} [Thu Aug 25 04:54:21 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:21.865300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd2265-6f09-42f7-81ca-8081212f033b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:21.891827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:21.971658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd2265-6f09-42f7-81ca-8081212f033b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:22.006840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da66b17c-fb80-468b-9af4-bbe8895b8fb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:22.007455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66b17c-fb80-468b-9af4-bbe8895b8fb4 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:22.007787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66b17c-fb80-468b-9af4-bbe8895b8fb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:22.008064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66b17c-fb80-468b-9af4-bbe8895b8fb4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:22.008720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da66b17c-fb80-468b-9af4-bbe8895b8fb4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:22.010186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da66b17c-fb80-468b-9af4-bbe8895b8fb4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:22.078371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 504/1010] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:54:21 2022] POST /identity/v3/endpoints => generated 325 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:54:23.153902 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be307412-5b82-4441-ba82-86f835ddd355 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:23.154437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be307412-5b82-4441-ba82-86f835ddd355 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:23.154437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be307412-5b82-4441-ba82-86f835ddd355 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:23.156032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 507/1011] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:23.161733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8675b-1af5-44a2-8a76-c1716fa13dad None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:23.162196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8675b-1af5-44a2-8a76-c1716fa13dad None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:23.162297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8675b-1af5-44a2-8a76-c1716fa13dad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:23.180570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-beb8675b-1af5-44a2-8a76-c1716fa13dad None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:23.181780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-beb8675b-1af5-44a2-8a76-c1716fa13dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:23.260274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 505/1012] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:23 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:23.527180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be307412-5b82-4441-ba82-86f835ddd355 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:23.636916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be307412-5b82-4441-ba82-86f835ddd355 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:23.644332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:23.671838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d99fd5-464f-4012-a71a-cae896703f45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:23.672473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d99fd5-464f-4012-a71a-cae896703f45 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:23.672751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d99fd5-464f-4012-a71a-cae896703f45 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:23.673018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d99fd5-464f-4012-a71a-cae896703f45 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:23.673621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d99fd5-464f-4012-a71a-cae896703f45 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:23.674976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d99fd5-464f-4012-a71a-cae896703f45 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-35d99fd5-464f-4012-a71a-cae896703f45 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:23.684447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:23.685469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 25 04:54:23.686649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:23.687403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5567b-4db1-4620-b54d-3f5cc12df68d None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:23.687403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5567b-4db1-4620-b54d-3f5cc12df68d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:23.687403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d5567b-4db1-4620-b54d-3f5cc12df68d None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:23.688027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 506/1014] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:23.689658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 508/1014] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:23 2022] GET /identity/v3/services/compute => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:23.691641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d5567b-4db1-4620-b54d-3f5cc12df68d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:23.692883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62ba7ab-9566-4a8b-8d4f-1b3b5d7b55fa None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:23.693171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62ba7ab-9566-4a8b-8d4f-1b3b5d7b55fa None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:23.693444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62ba7ab-9566-4a8b-8d4f-1b3b5d7b55fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:23.702693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:23.712193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d62ba7ab-9566-4a8b-8d4f-1b3b5d7b55fa None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:23.713373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d62ba7ab-9566-4a8b-8d4f-1b3b5d7b55fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:23.785248 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 509/1015] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:23 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:23.802923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d5567b-4db1-4620-b54d-3f5cc12df68d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:23.838311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8ed5d9-9688-47ff-9018-9249ed57be7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:23.838920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ed5d9-9688-47ff-9018-9249ed57be7d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:23.839200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ed5d9-9688-47ff-9018-9249ed57be7d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:23.839477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ed5d9-9688-47ff-9018-9249ed57be7d None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:23.840100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8ed5d9-9688-47ff-9018-9249ed57be7d None admin] RBAC: Adding query filter params (name=compute) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:23.840404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8ed5d9-9688-47ff-9018-9249ed57be7d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:23.841794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8ed5d9-9688-47ff-9018-9249ed57be7d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:23.846904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 507/1016] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:54:23 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:23.852474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62ba7ab-9566-4a8b-8d4f-1b3b5d7b55fa None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:23.949736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:23.953306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62ba7ab-9566-4a8b-8d4f-1b3b5d7b55fa None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:23.988208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:23.988645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:23.988923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:23.989192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:23.989685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5aae9b-6c81-4902-b794-e535e7a5c6f5 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:23.989854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] RBAC: Adding query filter params (type=compute) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:23.990008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5aae9b-6c81-4902-b794-e535e7a5c6f5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:23.990159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:23.990272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5aae9b-6c81-4902-b794-e535e7a5c6f5 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:23.991524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:23.991859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 508/1017] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:24.000976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 510/1018] 10.222.0.16 () {56 vars in 1112 bytes} [Thu Aug 25 04:54:23 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:24.001124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed94f5-b523-417a-9fbd-b5c8ae1365d0 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.001124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed94f5-b523-417a-9fbd-b5c8ae1365d0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.001124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eed94f5-b523-417a-9fbd-b5c8ae1365d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.002842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:24.015395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3eed94f5-b523-417a-9fbd-b5c8ae1365d0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:24.016581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3eed94f5-b523-417a-9fbd-b5c8ae1365d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:24.086181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 509/1019] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:23 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:24.087186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eed94f5-b523-417a-9fbd-b5c8ae1365d0 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:24.089330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:24.118598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5370e7-3cec-497f-8490-1e3580b2b13d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:24.155262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31bbac2-9568-4e14-bd11-fc2a628254fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:24.155916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31bbac2-9568-4e14-bd11-fc2a628254fd None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.156132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31bbac2-9568-4e14-bd11-fc2a628254fd None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.156405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31bbac2-9568-4e14-bd11-fc2a628254fd None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.157059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31bbac2-9568-4e14-bd11-fc2a628254fd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:24.158443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31bbac2-9568-4e14-bd11-fc2a628254fd None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:24.192715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eed94f5-b523-417a-9fbd-b5c8ae1365d0 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:24.228512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:24.229247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.229527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.229821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] PATH_INFO: `/v3/services/image` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.230402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:24.231871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:24.240322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 511/1020] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:54:24 2022] POST /identity/v3/endpoints => generated 334 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:24.240733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:24.241765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:24.242656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 510/1021] 10.222.0.16 () {56 vars in 1105 bytes} [Thu Aug 25 04:54:24 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:24.251935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31bbac2-9568-4e14-bd11-fc2a628254fd None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:24.278158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:24.357922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:24.373229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31bbac2-9568-4e14-bd11-fc2a628254fd None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:24.420425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:24.421176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.421461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.421767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.422404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] RBAC: Adding query filter params (name=image) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:24.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:24.424070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:24.429291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 512/1022] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:24 2022] GET /identity/v3/services?name=image => generated 122 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:24.435010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:24.435906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:24.473969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb0625d-5df8-43e7-9582-8e565309b31a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:24.513738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5ceba6-3093-45cd-8a95-7562c9e6ef9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:24.514627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5ceba6-3093-45cd-8a95-7562c9e6ef9a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.514943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5ceba6-3093-45cd-8a95-7562c9e6ef9a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.515169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5ceba6-3093-45cd-8a95-7562c9e6ef9a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.516277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5ceba6-3093-45cd-8a95-7562c9e6ef9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:24.517802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5ceba6-3093-45cd-8a95-7562c9e6ef9a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:24.519520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 511/1023] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:24.526720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5ceba6-3093-45cd-8a95-7562c9e6ef9a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:24.535449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:24.554657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa9b422-4b64-4725-bd58-649dc860b2f4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:24.594617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:24.595226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.595506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.595778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.596406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] RBAC: Adding query filter params (type=image) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:24.596709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:24.598111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:24.607760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 513/1024] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:24 2022] GET /identity/v3/services?type=image => generated 350 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:24.609503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:24.632182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5ceba6-3093-45cd-8a95-7562c9e6ef9a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:24.669233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bde67c-493c-4bd0-9fdc-c100a5c1a038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:24.669825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bde67c-493c-4bd0-9fdc-c100a5c1a038 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.670105 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bde67c-493c-4bd0-9fdc-c100a5c1a038 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.670375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bde67c-493c-4bd0-9fdc-c100a5c1a038 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.671006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bde67c-493c-4bd0-9fdc-c100a5c1a038 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:24.672435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bde67c-493c-4bd0-9fdc-c100a5c1a038 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:24.677275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:24.713865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659c63b7-2f6c-41e2-b789-fd5c753126c3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:24.747600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:24.748210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.748492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.748765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.749460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] RBAC: Adding query filter params (interface=public, service_id=894ca7b865674327a9c057fb3ed22251, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:24.749800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:24.751157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:24.756736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 514/1025] 10.222.0.16 () {56 vars in 1251 bytes} [Thu Aug 25 04:54:24 2022] GET /identity/v3/endpoints?service_id=894ca7b865674327a9c057fb3ed22251&interface=public®ion_id=RegionOne => generated 194 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:24.761043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 512/1026] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:54:24 2022] POST /identity/v3/projects => generated 311 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:24.762213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:24.774700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:24.872724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b96939-79b2-48bd-bd7a-acb9b9466f4c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:24.910893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55714f6b-2d27-436d-a2a6-ddcb00c0f8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:24.911490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55714f6b-2d27-436d-a2a6-ddcb00c0f8ce None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:24.911772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55714f6b-2d27-436d-a2a6-ddcb00c0f8ce None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:24.912041 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55714f6b-2d27-436d-a2a6-ddcb00c0f8ce None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:24.913198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55714f6b-2d27-436d-a2a6-ddcb00c0f8ce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:24.914597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55714f6b-2d27-436d-a2a6-ddcb00c0f8ce None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:24.921414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 515/1027] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:54:24 2022] GET /identity/v3/services => generated 1957 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 04:54:26.238295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a91765-283f-4630-806e-fcf5e7b8adb4 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:26.238779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a91765-283f-4630-806e-fcf5e7b8adb4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:26.238888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a91765-283f-4630-806e-fcf5e7b8adb4 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:26.240199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 513/1028] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:26.245538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df255c0-5ec1-4bcb-99a5-c5b7ae7e7790 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:26.245987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df255c0-5ec1-4bcb-99a5-c5b7ae7e7790 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:26.246170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df255c0-5ec1-4bcb-99a5-c5b7ae7e7790 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:26.264600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8df255c0-5ec1-4bcb-99a5-c5b7ae7e7790 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:26.265801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8df255c0-5ec1-4bcb-99a5-c5b7ae7e7790 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:26.336138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 516/1029] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:26 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:26.604266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a91765-283f-4630-806e-fcf5e7b8adb4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:26.698769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726d593-6921-4b3f-a56c-61ff09e24224 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:26.699123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726d593-6921-4b3f-a56c-61ff09e24224 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:26.699313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5726d593-6921-4b3f-a56c-61ff09e24224 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:26.700661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 517/1030] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:26.705869 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5dc370-9f0b-4e69-8ec0-28fe017af49c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:26.706178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5dc370-9f0b-4e69-8ec0-28fe017af49c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:26.706449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5dc370-9f0b-4e69-8ec0-28fe017af49c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:26.713505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a91765-283f-4630-806e-fcf5e7b8adb4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:26.713671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:26.727060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7f5dc370-9f0b-4e69-8ec0-28fe017af49c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:26.728238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7f5dc370-9f0b-4e69-8ec0-28fe017af49c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:26.749131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c0b40f-971a-49f7-a3d6-8c034eb62ee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:26.749809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c0b40f-971a-49f7-a3d6-8c034eb62ee6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:26.750087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c0b40f-971a-49f7-a3d6-8c034eb62ee6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:26.750357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c0b40f-971a-49f7-a3d6-8c034eb62ee6 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:26.762017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c0b40f-971a-49f7-a3d6-8c034eb62ee6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-c4c0b40f-971a-49f7-a3d6-8c034eb62ee6 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:26.766797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:26.767737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:26.770556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 514/1031] 10.222.0.16 () {56 vars in 1120 bytes} [Thu Aug 25 04:54:26 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:26.774334 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c0b40f-971a-49f7-a3d6-8c034eb62ee6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:26.805757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 518/1032] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:26 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:26.814889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9bba1-abbe-4691-9393-ef58114ba24e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:26.815177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9bba1-abbe-4691-9393-ef58114ba24e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:26.815452 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9bba1-abbe-4691-9393-ef58114ba24e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:26.816893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 519/1033] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:26.822312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88391350-9982-4ceb-b87f-5bf2b91c6343 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:26.822597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88391350-9982-4ceb-b87f-5bf2b91c6343 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:26.822877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88391350-9982-4ceb-b87f-5bf2b91c6343 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:26.826690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:26.841661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-88391350-9982-4ceb-b87f-5bf2b91c6343 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:26.842992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-88391350-9982-4ceb-b87f-5bf2b91c6343 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:26.882218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c0b40f-971a-49f7-a3d6-8c034eb62ee6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:26.916015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:26.916617 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:26.917091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:26.917212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 520/1034] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:26 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:26.917294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:26.917895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:26.918199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:26.919569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:26.932167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 515/1035] 10.222.0.16 () {56 vars in 1115 bytes} [Thu Aug 25 04:54:26 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:26.933473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88391350-9982-4ceb-b87f-5bf2b91c6343 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:26.994887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:27.031397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88391350-9982-4ceb-b87f-5bf2b91c6343 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.065409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.066058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.066332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.066602 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.077631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.077759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.079085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:27.094639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:27.095621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:27.096544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:27.096544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:27.096544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:27.096544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 521/1036] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:27.101394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.185088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca423614-b551-4b48-a4ba-51d927bae508 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.194676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:27.206585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f9b396-ae3c-4422-a7ec-9ec26113e0e0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.222770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983bf689-17b6-4984-bc1b-47423db4c11c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.222770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983bf689-17b6-4984-bc1b-47423db4c11c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.222770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983bf689-17b6-4984-bc1b-47423db4c11c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.222770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983bf689-17b6-4984-bc1b-47423db4c11c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.223691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:27.223781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983bf689-17b6-4984-bc1b-47423db4c11c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.223781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983bf689-17b6-4984-bc1b-47423db4c11c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.224026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 516/1037] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:27.224800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983bf689-17b6-4984-bc1b-47423db4c11c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.244648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.245266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.245545 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.245848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.246493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:27.246795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.248148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.260253 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 522/1038] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:27.261028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.328233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983bf689-17b6-4984-bc1b-47423db4c11c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.358500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe33732-a23b-40f8-9a74-d50d6529a7c0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.361622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.362311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.362592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.362865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] PATH_INFO: `/v3/services/image` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.363443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.364802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:27.373977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:27.380983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:27.381867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 517/1039] 10.222.0.16 () {56 vars in 1105 bytes} [Thu Aug 25 04:54:27 2022] GET /identity/v3/services/image => generated 86 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:27.382688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.393183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19797889-1f6c-409b-94bc-b8963dde3c7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.393818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19797889-1f6c-409b-94bc-b8963dde3c7f None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.394100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19797889-1f6c-409b-94bc-b8963dde3c7f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.394378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19797889-1f6c-409b-94bc-b8963dde3c7f None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.395018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19797889-1f6c-409b-94bc-b8963dde3c7f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.396428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19797889-1f6c-409b-94bc-b8963dde3c7f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.418747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:27.473538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd78c1b4-82d9-428e-b147-fb7d65718115 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.506588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.507208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.507486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.507754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.514550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.515932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:27.528112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:27.529282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 518/1040] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:54:27 2022] GET /identity/v3/users/nova => generated 82 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:27.530621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.564539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 523/1041] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:54:27 2022] POST /identity/v3/users => generated 285 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:54:27.565353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19797889-1f6c-409b-94bc-b8963dde3c7f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.600593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:27.646249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:27.650128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac570e6f-2fcc-4880-82c5-a15807105f45 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.679794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19797889-1f6c-409b-94bc-b8963dde3c7f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.687261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.687889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.688170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.688855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.689088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] RBAC: Adding query filter params (name=image) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:27.689395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.690855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.697115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 519/1042] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:27 2022] GET /identity/v3/services?name=image => generated 122 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:27.702911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.720218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.720837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.721115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.721384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.722162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:27.722466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.723930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.745816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 524/1043] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:54:27 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:27.752465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.820803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3768625b-d489-4be3-96e7-c9ab11ae5a43 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.853276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:27.854873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.855482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.855765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.856036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.856659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] RBAC: Adding query filter params (type=image) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:27.856965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.858417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:27.863355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 520/1044] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:27 2022] GET /identity/v3/services?type=image => generated 350 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:27.866479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5664426-7aa8-4fe3-8528-03e5f582d982 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.868961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:27.897626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:27.904013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5072351b-ab06-4afc-906a-3d98096e1f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:27.904646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5072351b-ab06-4afc-906a-3d98096e1f49 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:27.904919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5072351b-ab06-4afc-906a-3d98096e1f49 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:27.905185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5072351b-ab06-4afc-906a-3d98096e1f49 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:27.973367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aaffbc-1981-4c0e-a145-d3180756c339 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:27.980114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5072351b-ab06-4afc-906a-3d98096e1f49 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:27.981485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5072351b-ab06-4afc-906a-3d98096e1f49 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:28.009300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03528f30-af25-41be-8b9e-2f12a4317878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:28.009938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03528f30-af25-41be-8b9e-2f12a4317878 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:28.010216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03528f30-af25-41be-8b9e-2f12a4317878 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:28.010562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03528f30-af25-41be-8b9e-2f12a4317878 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:28.011123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03528f30-af25-41be-8b9e-2f12a4317878 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:28.012476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03528f30-af25-41be-8b9e-2f12a4317878 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:28.070432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:28.082629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 525/1045] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:27 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 25 04:54:28.089139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5072351b-ab06-4afc-906a-3d98096e1f49 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:28.095585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 521/1046] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:54:27 2022] POST /identity/v3/endpoints => generated 327 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:54:28.198753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5072351b-ab06-4afc-906a-3d98096e1f49 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:28.248856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:28.250171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:28.250603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:28.250872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:28.262631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:28.264192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:28.279938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:28.281108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:28.282303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:28.282303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:28.282303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:28.282303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 526/1047] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:28.286720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03528f30-af25-41be-8b9e-2f12a4317878 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:28.372779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:28.390085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03528f30-af25-41be-8b9e-2f12a4317878 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:28.424278 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:28.424882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:28.425160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:28.425427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:28.426086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:28.426386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:28.427855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:28.439335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 522/1048] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:28.445416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:28.482486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:28.546740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05606a1d-8ece-472b-b9f1-aec70e2649a2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:28.584100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:28.585235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:28.585512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:28.585825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:28.592855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:28.594340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:28.605386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:28.606461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:28.607374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 527/1049] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:28.612397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:28.715164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d668c92-7872-4b59-b4ce-e434fa220bdc None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:28.748976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e12c60a-361b-451e-934d-c23eeb2810fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:28.749578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e12c60a-361b-451e-934d-c23eeb2810fb None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:28.749880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e12c60a-361b-451e-934d-c23eeb2810fb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:28.750150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e12c60a-361b-451e-934d-c23eeb2810fb None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:28.750832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e12c60a-361b-451e-934d-c23eeb2810fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:28.751135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e12c60a-361b-451e-934d-c23eeb2810fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:28.752547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e12c60a-361b-451e-934d-c23eeb2810fb None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:28.771092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 523/1050] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:54:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:28.771258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:28.868263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:28.874700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714e8778-42fc-41d2-96c6-37635318423a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:28.907689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be6deb4-64d6-4746-a843-031cccdf6106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:28.908287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6deb4-64d6-4746-a843-031cccdf6106 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:28.908563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6deb4-64d6-4746-a843-031cccdf6106 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:28.908840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be6deb4-64d6-4746-a843-031cccdf6106 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:28.909556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be6deb4-64d6-4746-a843-031cccdf6106 None admin] RBAC: Adding query filter params (role.id=55d39bc11a0048728a909c34d6389fc1, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:28.909889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be6deb4-64d6-4746-a843-031cccdf6106 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:28.911294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6be6deb4-64d6-4746-a843-031cccdf6106 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:28.925619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 528/1051] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:28 2022] GET /identity/v3/role_assignments?user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=55d39bc11a0048728a909c34d6389fc1 => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:29.530256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2685bd-bf25-4882-a8ab-1fafe9b5b14d None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:29.530765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2685bd-bf25-4882-a8ab-1fafe9b5b14d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:29.530765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2685bd-bf25-4882-a8ab-1fafe9b5b14d None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:29.532085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 524/1052] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:29.537274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092b8fa-a5ef-4889-82bc-a10200f63410 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:29.537847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092b8fa-a5ef-4889-82bc-a10200f63410 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:29.537847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7092b8fa-a5ef-4889-82bc-a10200f63410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:29.556148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7092b8fa-a5ef-4889-82bc-a10200f63410 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:29.557312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7092b8fa-a5ef-4889-82bc-a10200f63410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:29.626213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 529/1053] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:29.891014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2685bd-bf25-4882-a8ab-1fafe9b5b14d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:29.894962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:29.996996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2685bd-bf25-4882-a8ab-1fafe9b5b14d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:30.030157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd483a-cd9f-43aa-89d2-ef90174af71f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.030338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333019c4-0c15-45ce-b349-7ad94462c110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:30.030476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd483a-cd9f-43aa-89d2-ef90174af71f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.030703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd483a-cd9f-43aa-89d2-ef90174af71f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.030795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333019c4-0c15-45ce-b349-7ad94462c110 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.031081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333019c4-0c15-45ce-b349-7ad94462c110 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.031353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333019c4-0c15-45ce-b349-7ad94462c110 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.032039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 530/1054] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:54:30.036771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096b8b7-1abb-4611-8cfe-850e630d8749 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.037580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096b8b7-1abb-4611-8cfe-850e630d8749 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.037878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d096b8b7-1abb-4611-8cfe-850e630d8749 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.042266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333019c4-0c15-45ce-b349-7ad94462c110 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-333019c4-0c15-45ce-b349-7ad94462c110 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:30.046983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:30.047914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:30.048780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:30.050040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 525/1055] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:54:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:30.054410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333019c4-0c15-45ce-b349-7ad94462c110 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:30.056200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d096b8b7-1abb-4611-8cfe-850e630d8749 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:30.057361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d096b8b7-1abb-4611-8cfe-850e630d8749 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:30.122925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:30.133765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 531/1056] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:30.160186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333019c4-0c15-45ce-b349-7ad94462c110 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:30.192879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:30.193553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.193858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.194127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.194775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:30.195074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:30.196430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:30.208835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 526/1057] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:54:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:30.210750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d096b8b7-1abb-4611-8cfe-850e630d8749 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:30.233498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:30.312550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d096b8b7-1abb-4611-8cfe-850e630d8749 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:30.347122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8999e4ee-5c98-422e-acca-35d4563b0a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:30.347751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8999e4ee-5c98-422e-acca-35d4563b0a1d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.348028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8999e4ee-5c98-422e-acca-35d4563b0a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.348297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8999e4ee-5c98-422e-acca-35d4563b0a1d None admin] PATH_INFO: `/v3/users/51d61ec3ff6f4858a7a00a66d8d55368` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.390033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:30.461773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8999e4ee-5c98-422e-acca-35d4563b0a1d None admin] RBAC: Authorizing `identity:get_user(user_id=51d61ec3ff6f4858a7a00a66d8d55368)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:30.463167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8999e4ee-5c98-422e-acca-35d4563b0a1d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:30.469694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 532/1058] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:30 2022] GET /identity/v3/users/51d61ec3ff6f4858a7a00a66d8d55368 => generated 285 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:30.470674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8999e4ee-5c98-422e-acca-35d4563b0a1d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:30.493375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0948aeb6-3d33-4439-96db-7bd59108fac5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:30.520144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:30.529779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97459aa-1a94-4cad-93d7-41ea0970677f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:30.530363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97459aa-1a94-4cad-93d7-41ea0970677f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.530899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97459aa-1a94-4cad-93d7-41ea0970677f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.531138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97459aa-1a94-4cad-93d7-41ea0970677f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.551668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:30.573721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8999e4ee-5c98-422e-acca-35d4563b0a1d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:30.605998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97459aa-1a94-4cad-93d7-41ea0970677f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:30.607802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630c9960-6501-4a6b-8948-786693b63140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:30.608201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97459aa-1a94-4cad-93d7-41ea0970677f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:30.608409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c9960-6501-4a6b-8948-786693b63140 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.608687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c9960-6501-4a6b-8948-786693b63140 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.608960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c9960-6501-4a6b-8948-786693b63140 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.679984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630c9960-6501-4a6b-8948-786693b63140 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:30.681416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630c9960-6501-4a6b-8948-786693b63140 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:30.721234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 527/1059] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 25 04:54:30.726551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97459aa-1a94-4cad-93d7-41ea0970677f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:30.758604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:30.763265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:30.790667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 533/1060] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:54:30.791956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630c9960-6501-4a6b-8948-786693b63140 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:30.831698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97459aa-1a94-4cad-93d7-41ea0970677f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:30.866106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61e6319-acd2-4e53-bd28-0e2d5836f210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:30.866739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61e6319-acd2-4e53-bd28-0e2d5836f210 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.867017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61e6319-acd2-4e53-bd28-0e2d5836f210 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.867287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61e6319-acd2-4e53-bd28-0e2d5836f210 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.876991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61e6319-acd2-4e53-bd28-0e2d5836f210 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:30.878460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61e6319-acd2-4e53-bd28-0e2d5836f210 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:30.892172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630c9960-6501-4a6b-8948-786693b63140 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-b61e6319-acd2-4e53-bd28-0e2d5836f210 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:30.893034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:30.894056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:30.894964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:30.894964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:30.894964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:30.894964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 528/1061] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:30.895427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a8675-3902-4229-9819-db9b0c925c8b None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.895705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a8675-3902-4229-9819-db9b0c925c8b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.895976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236a8675-3902-4229-9819-db9b0c925c8b None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.897230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 529/1062] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:30.899854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a8675-3902-4229-9819-db9b0c925c8b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:30.928707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293ad227-4aed-4511-8cb3-c156437be1a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:30.929337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293ad227-4aed-4511-8cb3-c156437be1a5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.929637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293ad227-4aed-4511-8cb3-c156437be1a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.929909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293ad227-4aed-4511-8cb3-c156437be1a5 None admin] PATH_INFO: `/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.931016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293ad227-4aed-4511-8cb3-c156437be1a5 None admin] RBAC: Authorizing `identity:get_project(project_id=6be022c3c7c44b9fab3d8e9fd402e509)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:30.932434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293ad227-4aed-4511-8cb3-c156437be1a5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:30.934068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 534/1063] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:54:30 2022] GET /identity/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:30.935241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e1bd0-0a2e-45b7-9273-b0bf1dacd9e4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:30.935524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e1bd0-0a2e-45b7-9273-b0bf1dacd9e4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:30.935799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e1bd0-0a2e-45b7-9273-b0bf1dacd9e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:30.952634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2e4e1bd0-0a2e-45b7-9273-b0bf1dacd9e4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:30.953838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2e4e1bd0-0a2e-45b7-9273-b0bf1dacd9e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:30.978658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:30.995611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:31.001045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236a8675-3902-4229-9819-db9b0c925c8b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:31.018861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 535/1064] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:31.019668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4e1bd0-0a2e-45b7-9273-b0bf1dacd9e4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:31.046348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:31.046972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:31.047240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:31.047510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:31.048153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:31.048456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:31.049844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:31.073683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 530/1065] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:31.139712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4e1bd0-0a2e-45b7-9273-b0bf1dacd9e4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:31.175813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:31.176433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:31.176775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:31.177052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:31.178378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=6be022c3c7c44b9fab3d8e9fd402e509, user.id=51d61ec3ff6f4858a7a00a66d8d55368) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:31.178693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:31.180130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:31.189634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 536/1066] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:31 2022] GET /identity/v3/role_assignments?user.id=51d61ec3ff6f4858a7a00a66d8d55368&scope.project.id=6be022c3c7c44b9fab3d8e9fd402e509&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:31.289840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:31.396028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38ded47-de96-4fdc-a046-ccde19e9946d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:31.490991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110e91f8-b26a-486e-8b4e-51e850df9474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:31.491624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110e91f8-b26a-486e-8b4e-51e850df9474 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:31.491903 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110e91f8-b26a-486e-8b4e-51e850df9474 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:31.492173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110e91f8-b26a-486e-8b4e-51e850df9474 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:31.498776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:31.502489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110e91f8-b26a-486e-8b4e-51e850df9474 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-110e91f8-b26a-486e-8b4e-51e850df9474 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:31.507213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:31.514236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:31.515282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 531/1067] 10.222.0.16 () {56 vars in 1120 bytes} [Thu Aug 25 04:54:31 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:31.516173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:31.557566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:31.614385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1672e4f8-d9d1-4d69-9fec-002826b72d5c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:31.648395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:31.648994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:31.649266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:31.649532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:31.650214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:31.650511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:31.651852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:31.659791 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 537/1068] 10.222.0.16 () {56 vars in 1115 bytes} [Thu Aug 25 04:54:31 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:31.666546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110e91f8-b26a-486e-8b4e-51e850df9474 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:31.768343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110e91f8-b26a-486e-8b4e-51e850df9474 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:31.802984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:31.803609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:31.803888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:31.804159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:31.813964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:31.815347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:31.829372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:31.830529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:31.831483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:31.831483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:31.831483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:31.833751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 532/1069] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:31.837103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:31.934201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035b93ab-9485-43b1-beda-fd494b08c49b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:31.935685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:31.967206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:31.967796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:31.968071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:31.968340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:31.968968 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:31.969266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:31.970646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:31.982037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 538/1070] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:31.987634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:32.035681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:32.083054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b67111-70bf-48c0-ba2f-8862f866b4ea None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:32.115746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:32.116370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:32.116646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:32.116913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:32.124225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:32.125671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:32.136827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:32.138020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:32.139181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 533/1071] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:54:31 2022] GET /identity/v3/users/nova => generated 82 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:32.143765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:32.242055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc66ccee-0aa7-4fac-80f5-026bd06f13f0 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:32.274495 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:32.275095 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:32.275371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:32.275641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:32.276362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:32.276663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:32.278122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:32.294967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 539/1072] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:54:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:32.300605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:32.399113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9930558e-8fed-4bec-8191-29fd8a39de32 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:32.409102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:32.432250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:32.432872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:32.433151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:32.433420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:32.443592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:32.445029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:32.458625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:32.460213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:32.461131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:32.461131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:32.461131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:32.465123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 534/1073] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:32.466501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:32.488084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:32.566133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8da708-8bd5-452d-928f-64f990f51228 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:32.600745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:32.601347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:32.601649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:32.601926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:32.602564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:32.602865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:32.604227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:32.616019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 540/1074] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:32.621645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:32.718806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d6d584-de1d-4fcd-8d5b-5b4d6803a6a3 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:32.751289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:32.751912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:32.752193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:32.752465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:32.759574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:32.761009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:32.771969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:32.773038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:32.773987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 535/1075] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:32.779097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:32.864506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:32.883177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37f5465-e4c2-4bf4-a07e-7c3ac5d2a90c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:32.916792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e8d73f-acc0-4e30-bcd6-e38b0b4e3294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:32.917396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e8d73f-acc0-4e30-bcd6-e38b0b4e3294 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:32.917705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e8d73f-acc0-4e30-bcd6-e38b0b4e3294 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:32.917981 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e8d73f-acc0-4e30-bcd6-e38b0b4e3294 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:32.918668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e8d73f-acc0-4e30-bcd6-e38b0b4e3294 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:32.918970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e8d73f-acc0-4e30-bcd6-e38b0b4e3294 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:32.920375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e8d73f-acc0-4e30-bcd6-e38b0b4e3294 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:32.932380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 541/1076] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:54:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:32.938343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:32.962590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:33.037462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ee4ef7-b074-4126-b300-3c555c8aea20 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:33.070809 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6b65ec-3006-4db0-974f-6e15d7d0265f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:33.071466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6b65ec-3006-4db0-974f-6e15d7d0265f None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:33.071748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6b65ec-3006-4db0-974f-6e15d7d0265f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:33.072022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6b65ec-3006-4db0-974f-6e15d7d0265f None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/eb5ac6ef2f5048d3aed0d7a91abc4cac/roles/55d39bc11a0048728a909c34d6389fc1` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:33.074160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6b65ec-3006-4db0-974f-6e15d7d0265f None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=eb5ac6ef2f5048d3aed0d7a91abc4cac, role_id=55d39bc11a0048728a909c34d6389fc1)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:33.075588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6b65ec-3006-4db0-974f-6e15d7d0265f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:33.080227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a80962-df06-4285-9bc3-eee26dc60b03 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:33.080520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a80962-df06-4285-9bc3-eee26dc60b03 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:33.080797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a80962-df06-4285-9bc3-eee26dc60b03 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:33.082171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 542/1077] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:54:33.087459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a9d3b0-8d0b-4aff-b5fa-01e243ee5e74 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:33.087681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a9d3b0-8d0b-4aff-b5fa-01e243ee5e74 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:33.087964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a9d3b0-8d0b-4aff-b5fa-01e243ee5e74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:33.106112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e8a9d3b0-8d0b-4aff-b5fa-01e243ee5e74 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:33.107301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e8a9d3b0-8d0b-4aff-b5fa-01e243ee5e74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:33.164519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 536/1078] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:54:32 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/eb5ac6ef2f5048d3aed0d7a91abc4cac/roles/55d39bc11a0048728a909c34d6389fc1 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 04:54:33.177838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 543/1079] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:33.438862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6b65ec-3006-4db0-974f-6e15d7d0265f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:33.520098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:33.537862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6b65ec-3006-4db0-974f-6e15d7d0265f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:33.571111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dd77ba-a65c-4f31-92db-fa794f116ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:33.571751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dd77ba-a65c-4f31-92db-fa794f116ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:33.572019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dd77ba-a65c-4f31-92db-fa794f116ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:33.572288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dd77ba-a65c-4f31-92db-fa794f116ad6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:33.583084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dd77ba-a65c-4f31-92db-fa794f116ad6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-27dd77ba-a65c-4f31-92db-fa794f116ad6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:33.588685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:33.589830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:33.590703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 537/1080] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:54:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:33.595724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a9d3b0-8d0b-4aff-b5fa-01e243ee5e74 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:33.661310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:33.690262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a9d3b0-8d0b-4aff-b5fa-01e243ee5e74 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:33.723077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:33.723684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:33.723995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:33.724238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:33.724930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:33.725234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:33.726611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:33.734393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 544/1081] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:54:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:33.740285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dd77ba-a65c-4f31-92db-fa794f116ad6 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:33.837259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dd77ba-a65c-4f31-92db-fa794f116ad6 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:33.869734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be8c289-db5c-485f-9421-7435ec3ceb4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:33.870352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8c289-db5c-485f-9421-7435ec3ceb4c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:33.870629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8c289-db5c-485f-9421-7435ec3ceb4c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:33.870899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8c289-db5c-485f-9421-7435ec3ceb4c None admin] PATH_INFO: `/v3/users/51d61ec3ff6f4858a7a00a66d8d55368` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:33.872000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be8c289-db5c-485f-9421-7435ec3ceb4c None admin] RBAC: Authorizing `identity:get_user(user_id=51d61ec3ff6f4858a7a00a66d8d55368)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:33.873424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be8c289-db5c-485f-9421-7435ec3ceb4c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:33.875318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 538/1082] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:33 2022] GET /identity/v3/users/51d61ec3ff6f4858a7a00a66d8d55368 => generated 285 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:33.881230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:33.976533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64eb918-4c5f-40fe-81b2-3e7b9385b675 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:34.004210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:34.009554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2f622b-093a-40e0-8945-366e83b53cf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:34.010194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2f622b-093a-40e0-8945-366e83b53cf3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:34.010469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2f622b-093a-40e0-8945-366e83b53cf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:34.010738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2f622b-093a-40e0-8945-366e83b53cf3 None admin] PATH_INFO: `/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:34.011825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2f622b-093a-40e0-8945-366e83b53cf3 None admin] RBAC: Authorizing `identity:get_project(project_id=6be022c3c7c44b9fab3d8e9fd402e509)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:34.013233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2f622b-093a-40e0-8945-366e83b53cf3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:34.014871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 545/1083] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:54:33 2022] GET /identity/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:34.020720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be8c289-db5c-485f-9421-7435ec3ceb4c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:34.033376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:34.119908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be8c289-db5c-485f-9421-7435ec3ceb4c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:34.152920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b982c662-b8e1-437d-a231-dff7997b133f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:34.153581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982c662-b8e1-437d-a231-dff7997b133f None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:34.153889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982c662-b8e1-437d-a231-dff7997b133f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:34.154165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b982c662-b8e1-437d-a231-dff7997b133f None admin] PATH_INFO: `/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509/users/51d61ec3ff6f4858a7a00a66d8d55368/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:34.156060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982c662-b8e1-437d-a231-dff7997b133f None admin] RBAC: Authorizing `identity:create_grant(project_id=6be022c3c7c44b9fab3d8e9fd402e509, user_id=51d61ec3ff6f4858a7a00a66d8d55368, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:34.157490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b982c662-b8e1-437d-a231-dff7997b133f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:34.221095 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 539/1084] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:54:34 2022] PUT /identity/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509/users/51d61ec3ff6f4858a7a00a66d8d55368/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:54:35.036411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88120067-91e2-4654-842d-793f3ef62642 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:35.036938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88120067-91e2-4654-842d-793f3ef62642 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:35.037026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88120067-91e2-4654-842d-793f3ef62642 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:35.038375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 546/1085] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:35.043812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5872cac2-eb5e-46e7-a10d-367420aff9ab None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:35.044238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5872cac2-eb5e-46e7-a10d-367420aff9ab None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:35.044467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5872cac2-eb5e-46e7-a10d-367420aff9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:35.062078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5872cac2-eb5e-46e7-a10d-367420aff9ab None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:35.063259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5872cac2-eb5e-46e7-a10d-367420aff9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:35.123933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:35.128375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 540/1086] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:35.395727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88120067-91e2-4654-842d-793f3ef62642 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:35.493966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88120067-91e2-4654-842d-793f3ef62642 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:35.526660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4906d4d-c8ae-4de4-bff7-2ebbf4acd166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:35.527278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4906d4d-c8ae-4de4-bff7-2ebbf4acd166 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:35.527551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4906d4d-c8ae-4de4-bff7-2ebbf4acd166 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:35.527814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4906d4d-c8ae-4de4-bff7-2ebbf4acd166 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:35.537675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4906d4d-c8ae-4de4-bff7-2ebbf4acd166 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-a4906d4d-c8ae-4de4-bff7-2ebbf4acd166 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:35.542294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:35.545796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:35.546897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 547/1087] 10.222.0.16 () {56 vars in 1120 bytes} [Thu Aug 25 04:54:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:35.549208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5872cac2-eb5e-46e7-a10d-367420aff9ab None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:35.645911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5872cac2-eb5e-46e7-a10d-367420aff9ab None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:35.678484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:35.679081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:35.679355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:35.679623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:35.680270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:35.680568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:35.681963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:35.689651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 541/1088] 10.222.0.16 () {56 vars in 1115 bytes} [Thu Aug 25 04:54:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:35.695989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4906d4d-c8ae-4de4-bff7-2ebbf4acd166 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:35.753896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:35.793651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4906d4d-c8ae-4de4-bff7-2ebbf4acd166 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:35.826933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:35.827557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:35.827834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:35.828105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:35.838621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:35.840042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:35.854104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:35.860157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:35.861166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:35.861166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:35.861166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:35.861166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 548/1089] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:35.861394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:35.920763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:35.958888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a83f365-9323-4305-9376-2e65a84c5857 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:35.991506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d19001-52be-4f57-96b6-821cb3b54f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:35.992110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d19001-52be-4f57-96b6-821cb3b54f65 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:35.992385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d19001-52be-4f57-96b6-821cb3b54f65 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:35.992653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d19001-52be-4f57-96b6-821cb3b54f65 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:35.993283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d19001-52be-4f57-96b6-821cb3b54f65 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:35.993580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d19001-52be-4f57-96b6-821cb3b54f65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:35.994956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d19001-52be-4f57-96b6-821cb3b54f65 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:36.006935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 542/1090] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:36.012591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.109732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ffd6f-fcbc-419e-8d2e-6c6c577f9b4f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:36.122478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:36.135763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10a893-3598-48e5-9057-685eb7816f78 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.136252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10a893-3598-48e5-9057-685eb7816f78 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.136349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10a893-3598-48e5-9057-685eb7816f78 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.137624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 543/1091] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:54:36.142621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fb10e-3ea2-4cdc-8a34-1392c1d98674 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.142904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fb10e-3ea2-4cdc-8a34-1392c1d98674 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.143176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fb10e-3ea2-4cdc-8a34-1392c1d98674 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.143486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:36.144211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.144492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.144789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] PATH_INFO: `/v3/users/nova` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.152848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:36.154530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:36.160313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-847fb10e-3ea2-4cdc-8a34-1392c1d98674 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:36.161504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-847fb10e-3ea2-4cdc-8a34-1392c1d98674 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:54:36.165480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:36.166674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:36.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 549/1092] 10.222.0.16 () {56 vars in 1093 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:36.172235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.229884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 544/1093] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:36.276280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f814603-5778-4a76-8c92-de88ee7d955c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:36.308147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:36.308741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.309015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.309279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.310031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:36.310332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:36.311730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:36.328018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 550/1094] 10.222.0.16 () {56 vars in 1133 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:36.334214 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847fb10e-3ea2-4cdc-8a34-1392c1d98674 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.386836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:36.432350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847fb10e-3ea2-4cdc-8a34-1392c1d98674 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:36.465064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9f8d5d-1dc3-41c9-9c4b-b048bdd67633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:36.465690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9f8d5d-1dc3-41c9-9c4b-b048bdd67633 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.465969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9f8d5d-1dc3-41c9-9c4b-b048bdd67633 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.466237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9f8d5d-1dc3-41c9-9c4b-b048bdd67633 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.498922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.529294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:36.531646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9f8d5d-1dc3-41c9-9c4b-b048bdd67633 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:36.533091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9f8d5d-1dc3-41c9-9c4b-b048bdd67633 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:36.583693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:36.596223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca03b7c-c0ad-43ae-aacb-29d25c9de962 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:36.630642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1027b469-b449-4447-9b8d-bfb896a8ea23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:36.631169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 545/1095] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:54:36.631322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027b469-b449-4447-9b8d-bfb896a8ea23 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.631529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027b469-b449-4447-9b8d-bfb896a8ea23 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.631801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1027b469-b449-4447-9b8d-bfb896a8ea23 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.636526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9f8d5d-1dc3-41c9-9c4b-b048bdd67633 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.641955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1027b469-b449-4447-9b8d-bfb896a8ea23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1027b469-b449-4447-9b8d-bfb896a8ea23 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:36.646651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:36.647728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:54:36.648633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 551/1096] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:36.653396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1027b469-b449-4447-9b8d-bfb896a8ea23 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.736406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9f8d5d-1dc3-41c9-9c4b-b048bdd67633 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:36.742046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:36.751313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1027b469-b449-4447-9b8d-bfb896a8ea23 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:36.770040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:36.770660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.770934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.771759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.781583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:36.783035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:36.785503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:36.786139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.786417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.786686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.787333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:36.787630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:36.788955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:36.796781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 552/1097] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:54:36.797104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:36.798168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:54:36.799099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:54:36.799099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:54:36.799099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:36.799099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 546/1098] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 25 04:54:36.802747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.803848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.807251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:36.902196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b826c9-ce39-4661-b754-ca2cfc9d2a09 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:36.902507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d563884b-958d-4abc-af9f-06d79eb75cee None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:36.935494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb62efda-9224-453b-80bf-8a2fe8623475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:36.935677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:36.936114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62efda-9224-453b-80bf-8a2fe8623475 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.936216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:36.936386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62efda-9224-453b-80bf-8a2fe8623475 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.936492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:36.936647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62efda-9224-453b-80bf-8a2fe8623475 None admin] PATH_INFO: `/v3/users/51d61ec3ff6f4858a7a00a66d8d55368` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.936797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:36.937399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:36.937743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:36.937851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb62efda-9224-453b-80bf-8a2fe8623475 None admin] RBAC: Authorizing `identity:get_user(user_id=51d61ec3ff6f4858a7a00a66d8d55368)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:36.939113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:36.939215 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb62efda-9224-453b-80bf-8a2fe8623475 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:36.940713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 553/1099] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/users/51d61ec3ff6f4858a7a00a66d8d55368 => generated 285 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:36.947268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb62efda-9224-453b-80bf-8a2fe8623475 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.951888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 547/1100] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:36.957257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:36.969225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:37.018431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:37.045890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb62efda-9224-453b-80bf-8a2fe8623475 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:37.055899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6763090-65cd-4167-a42b-427eb013fe27 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:37.079419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd65adc-979e-4ade-b425-e719f8618b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:37.080016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd65adc-979e-4ade-b425-e719f8618b9d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:37.080290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd65adc-979e-4ade-b425-e719f8618b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:37.080558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd65adc-979e-4ade-b425-e719f8618b9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:37.089022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:37.089658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:37.089935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:37.090202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:37.096662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:37.098136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:54:37.110153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:37.111070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:37.111978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 548/1101] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:37.117047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:37.147025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd65adc-979e-4ade-b425-e719f8618b9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:37.148530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd65adc-979e-4ade-b425-e719f8618b9d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:37.168674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:37.216358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c30eaac-944b-4f41-a96e-9c8f306f4f52 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:37.242846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:37.247554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 554/1102] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 25 04:54:37.253252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:37.253400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd65adc-979e-4ade-b425-e719f8618b9d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:37.263659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:37.263659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:37.263659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:37.263659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:37.263659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:37.263659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:37.269479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 549/1103] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:54:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:37.275588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:37.355148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd65adc-979e-4ade-b425-e719f8618b9d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:37.372730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca1907b-58c4-4dcb-a622-459275b0f10c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:37.374800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:37.388999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80b62ed-37bb-4ce2-b2fb-fb8b0f7a8fb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:37.389640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80b62ed-37bb-4ce2-b2fb-fb8b0f7a8fb8 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:37.389917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80b62ed-37bb-4ce2-b2fb-fb8b0f7a8fb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:37.390183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80b62ed-37bb-4ce2-b2fb-fb8b0f7a8fb8 None admin] PATH_INFO: `/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:37.391284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80b62ed-37bb-4ce2-b2fb-fb8b0f7a8fb8 None admin] RBAC: Authorizing `identity:get_project(project_id=6be022c3c7c44b9fab3d8e9fd402e509)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:37.392728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80b62ed-37bb-4ce2-b2fb-fb8b0f7a8fb8 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:37.394522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 555/1104] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:54:37 2022] GET /identity/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:37.400489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80b62ed-37bb-4ce2-b2fb-fb8b0f7a8fb8 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:37.411019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfdc743-3c9f-43e4-b1da-58051a92f8e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:37.412267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfdc743-3c9f-43e4-b1da-58051a92f8e7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:37.412575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfdc743-3c9f-43e4-b1da-58051a92f8e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:37.412849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfdc743-3c9f-43e4-b1da-58051a92f8e7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:37.413568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfdc743-3c9f-43e4-b1da-58051a92f8e7 None admin] RBAC: Adding query filter params (role.id=55d39bc11a0048728a909c34d6389fc1, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:37.413908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfdc743-3c9f-43e4-b1da-58051a92f8e7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:37.415450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfdc743-3c9f-43e4-b1da-58051a92f8e7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:37.421239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 550/1105] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:37 2022] GET /identity/v3/role_assignments?user.id=eb5ac6ef2f5048d3aed0d7a91abc4cac&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=55d39bc11a0048728a909c34d6389fc1 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:37.504252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80b62ed-37bb-4ce2-b2fb-fb8b0f7a8fb8 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:37.538882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e976c62-0f15-46d2-98c2-d2c7a769b86a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:37.539411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e976c62-0f15-46d2-98c2-d2c7a769b86a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:37.539685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e976c62-0f15-46d2-98c2-d2c7a769b86a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:37.539952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e976c62-0f15-46d2-98c2-d2c7a769b86a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:37.540675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e976c62-0f15-46d2-98c2-d2c7a769b86a None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=6be022c3c7c44b9fab3d8e9fd402e509, user.id=51d61ec3ff6f4858a7a00a66d8d55368) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:37.540978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e976c62-0f15-46d2-98c2-d2c7a769b86a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:37.542434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e976c62-0f15-46d2-98c2-d2c7a769b86a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:37.548688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 556/1106] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:37 2022] GET /identity/v3/role_assignments?user.id=51d61ec3ff6f4858a7a00a66d8d55368&scope.project.id=6be022c3c7c44b9fab3d8e9fd402e509&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:39.489443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f75a392-a2e9-4d2c-88cc-fba22df0c25a None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:39.490070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f75a392-a2e9-4d2c-88cc-fba22df0c25a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:39.490070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f75a392-a2e9-4d2c-88cc-fba22df0c25a None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:39.491326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 551/1107] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:39.496631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79bd15-6ba9-4317-8dcd-84b0ab28e5e1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:39.497068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79bd15-6ba9-4317-8dcd-84b0ab28e5e1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:39.497167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79bd15-6ba9-4317-8dcd-84b0ab28e5e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:39.515154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0e79bd15-6ba9-4317-8dcd-84b0ab28e5e1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:39.516319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0e79bd15-6ba9-4317-8dcd-84b0ab28e5e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:39.560714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:39.579533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 557/1108] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:39.847327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f75a392-a2e9-4d2c-88cc-fba22df0c25a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:39.896667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:39.946129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f75a392-a2e9-4d2c-88cc-fba22df0c25a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:39.978673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55563c6a-6b85-4688-93fc-a7449e41dc03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:39.979287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55563c6a-6b85-4688-93fc-a7449e41dc03 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:39.979563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55563c6a-6b85-4688-93fc-a7449e41dc03 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:39.979831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55563c6a-6b85-4688-93fc-a7449e41dc03 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:39.980913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55563c6a-6b85-4688-93fc-a7449e41dc03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:39.982349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55563c6a-6b85-4688-93fc-a7449e41dc03 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:39.983864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 552/1109] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:39.989651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79bd15-6ba9-4317-8dcd-84b0ab28e5e1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:40.085268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79bd15-6ba9-4317-8dcd-84b0ab28e5e1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:40.118414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef2cb02-31b9-4e1c-a235-d6301657f887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:40.118952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef2cb02-31b9-4e1c-a235-d6301657f887 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:40.119229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef2cb02-31b9-4e1c-a235-d6301657f887 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:40.119501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef2cb02-31b9-4e1c-a235-d6301657f887 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:40.120141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef2cb02-31b9-4e1c-a235-d6301657f887 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:40.121554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef2cb02-31b9-4e1c-a235-d6301657f887 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:40.243878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 558/1110] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:54:39 2022] POST /identity/v3/users => generated 286 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 04:54:42.160181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903af1c1-b3af-4d20-a38b-de1f41792df9 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:42.160735 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903af1c1-b3af-4d20-a38b-de1f41792df9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:42.160735 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903af1c1-b3af-4d20-a38b-de1f41792df9 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:42.162183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 553/1111] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:42.167404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc9c19-8f5d-4496-9639-18e2d048163f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:42.167786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc9c19-8f5d-4496-9639-18e2d048163f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:42.167956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc9c19-8f5d-4496-9639-18e2d048163f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:42.186751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-60bc9c19-8f5d-4496-9639-18e2d048163f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:42.188601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-60bc9c19-8f5d-4496-9639-18e2d048163f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:42.218483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:42.254512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 559/1112] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:42.531801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903af1c1-b3af-4d20-a38b-de1f41792df9 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:42.628964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903af1c1-b3af-4d20-a38b-de1f41792df9 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:42.645142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:42.662327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e7ba75-6acf-4afb-848e-57edb9e81306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:42.662942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e7ba75-6acf-4afb-848e-57edb9e81306 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:42.663217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e7ba75-6acf-4afb-848e-57edb9e81306 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:42.663483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e7ba75-6acf-4afb-848e-57edb9e81306 None admin] PATH_INFO: `/v3/roles/609ec30d147941b4a2b5508980aea7d0` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:42.664842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e7ba75-6acf-4afb-848e-57edb9e81306 None admin] RBAC: Authorizing `identity:get_role(role_id=609ec30d147941b4a2b5508980aea7d0)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:42.666289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e7ba75-6acf-4afb-848e-57edb9e81306 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:42.667564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 554/1113] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:42 2022] GET /identity/v3/roles/609ec30d147941b4a2b5508980aea7d0 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:54:42.673264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc9c19-8f5d-4496-9639-18e2d048163f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:42.768306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc9c19-8f5d-4496-9639-18e2d048163f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:42.800928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29defbd6-1373-4f3f-8738-4938b665d6b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:42.801473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29defbd6-1373-4f3f-8738-4938b665d6b3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:42.801778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29defbd6-1373-4f3f-8738-4938b665d6b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:42.802051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29defbd6-1373-4f3f-8738-4938b665d6b3 None admin] PATH_INFO: `/v3/users/b227b1d78f44468f8f4edbd8cb4db514` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:42.815065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29defbd6-1373-4f3f-8738-4938b665d6b3 None admin] RBAC: Authorizing `identity:get_user(user_id=b227b1d78f44468f8f4edbd8cb4db514)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:42.816499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29defbd6-1373-4f3f-8738-4938b665d6b3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:42.818651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 560/1114] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:42 2022] GET /identity/v3/users/b227b1d78f44468f8f4edbd8cb4db514 => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:42.824213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e7ba75-6acf-4afb-848e-57edb9e81306 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:42.921035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e7ba75-6acf-4afb-848e-57edb9e81306 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:42.954007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a954bd-02ab-40df-8a56-f273ba66b32d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:42.954597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a954bd-02ab-40df-8a56-f273ba66b32d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:42.954873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a954bd-02ab-40df-8a56-f273ba66b32d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:42.955142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a954bd-02ab-40df-8a56-f273ba66b32d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:43.004175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:43.019642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a954bd-02ab-40df-8a56-f273ba66b32d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:43.021209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a954bd-02ab-40df-8a56-f273ba66b32d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:43.115873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 555/1115] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:42 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:54:43.121750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29defbd6-1373-4f3f-8738-4938b665d6b3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:43.148377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:43.216847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29defbd6-1373-4f3f-8738-4938b665d6b3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:43.250276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371cc3f9-b951-4346-99ff-13458bd0fd6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:43.250900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371cc3f9-b951-4346-99ff-13458bd0fd6c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:43.251176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371cc3f9-b951-4346-99ff-13458bd0fd6c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:43.251446 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371cc3f9-b951-4346-99ff-13458bd0fd6c None admin] PATH_INFO: `/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:43.252548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371cc3f9-b951-4346-99ff-13458bd0fd6c None admin] RBAC: Authorizing `identity:get_project(project_id=6be022c3c7c44b9fab3d8e9fd402e509)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:43.254002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371cc3f9-b951-4346-99ff-13458bd0fd6c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:43.256241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 561/1116] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:54:43 2022] GET /identity/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:43.262373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a954bd-02ab-40df-8a56-f273ba66b32d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:43.343094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:43.358524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a954bd-02ab-40df-8a56-f273ba66b32d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:43.391324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba62b1c5-c474-4597-b434-200cd15dd4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:43.391912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba62b1c5-c474-4597-b434-200cd15dd4b6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:43.392193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba62b1c5-c474-4597-b434-200cd15dd4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:43.392464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba62b1c5-c474-4597-b434-200cd15dd4b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:43.393171 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba62b1c5-c474-4597-b434-200cd15dd4b6 None admin] RBAC: Adding query filter params (role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=6be022c3c7c44b9fab3d8e9fd402e509, user.id=b227b1d78f44468f8f4edbd8cb4db514) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:43.393479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba62b1c5-c474-4597-b434-200cd15dd4b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:43.394922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba62b1c5-c474-4597-b434-200cd15dd4b6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:43.400090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 556/1117] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:43 2022] GET /identity/v3/role_assignments?user.id=b227b1d78f44468f8f4edbd8cb4db514&scope.project.id=6be022c3c7c44b9fab3d8e9fd402e509&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:45.238603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a15154c-9b53-4443-a45f-56538117e5ed None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:45.239026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a15154c-9b53-4443-a45f-56538117e5ed None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:45.239156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a15154c-9b53-4443-a45f-56538117e5ed None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:45.241017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 562/1118] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:45.246085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d1493-fc84-4ccb-8558-6075cd19e92a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:45.246517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d1493-fc84-4ccb-8558-6075cd19e92a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:45.246644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6d1493-fc84-4ccb-8558-6075cd19e92a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:45.264793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-9f6d1493-fc84-4ccb-8558-6075cd19e92a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:45.266003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9f6d1493-fc84-4ccb-8558-6075cd19e92a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:45.330402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 557/1119] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 25 04:54:45.595864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a15154c-9b53-4443-a45f-56538117e5ed None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:45.686615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:45.693802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a15154c-9b53-4443-a45f-56538117e5ed None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:45.726753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec71ba6-25f2-4602-9552-94eadf3cc335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:45.727363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec71ba6-25f2-4602-9552-94eadf3cc335 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:45.727637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec71ba6-25f2-4602-9552-94eadf3cc335 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:45.727904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec71ba6-25f2-4602-9552-94eadf3cc335 None admin] PATH_INFO: `/v3/roles/609ec30d147941b4a2b5508980aea7d0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:45.729255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec71ba6-25f2-4602-9552-94eadf3cc335 None admin] RBAC: Authorizing `identity:get_role(role_id=609ec30d147941b4a2b5508980aea7d0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:45.730628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec71ba6-25f2-4602-9552-94eadf3cc335 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:45.731915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 563/1120] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:45 2022] GET /identity/v3/roles/609ec30d147941b4a2b5508980aea7d0 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:45.737785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6d1493-fc84-4ccb-8558-6075cd19e92a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:45.812858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:45.832987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6d1493-fc84-4ccb-8558-6075cd19e92a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:45.868458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b56af-b736-4eaa-a622-353b60fc5cbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:45.869026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b56af-b736-4eaa-a622-353b60fc5cbb None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:45.869307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b56af-b736-4eaa-a622-353b60fc5cbb None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:45.869615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b56af-b736-4eaa-a622-353b60fc5cbb None admin] PATH_INFO: `/v3/users/b227b1d78f44468f8f4edbd8cb4db514` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:45.870728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710b56af-b736-4eaa-a622-353b60fc5cbb None admin] RBAC: Authorizing `identity:get_user(user_id=b227b1d78f44468f8f4edbd8cb4db514)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:45.872195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710b56af-b736-4eaa-a622-353b60fc5cbb None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:45.873793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 558/1121] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:45 2022] GET /identity/v3/users/b227b1d78f44468f8f4edbd8cb4db514 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:45.879483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec71ba6-25f2-4602-9552-94eadf3cc335 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:45.975188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec71ba6-25f2-4602-9552-94eadf3cc335 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:46.007299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015c925a-5e3b-4d91-8920-9eaa1fa7cc87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:46.007911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015c925a-5e3b-4d91-8920-9eaa1fa7cc87 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:46.008190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015c925a-5e3b-4d91-8920-9eaa1fa7cc87 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:46.008462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015c925a-5e3b-4d91-8920-9eaa1fa7cc87 None admin] PATH_INFO: `/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:46.009566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015c925a-5e3b-4d91-8920-9eaa1fa7cc87 None admin] RBAC: Authorizing `identity:get_project(project_id=6be022c3c7c44b9fab3d8e9fd402e509)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:46.011008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015c925a-5e3b-4d91-8920-9eaa1fa7cc87 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:46.012613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 564/1122] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:54:45 2022] GET /identity/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:46.018272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b56af-b736-4eaa-a622-353b60fc5cbb None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:46.115426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b56af-b736-4eaa-a622-353b60fc5cbb None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:46.147612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ae8fa5-9e97-4121-a376-638883f94a5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:46.148251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ae8fa5-9e97-4121-a376-638883f94a5d None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:46.148528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ae8fa5-9e97-4121-a376-638883f94a5d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:46.148798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ae8fa5-9e97-4121-a376-638883f94a5d None admin] PATH_INFO: `/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509/users/b227b1d78f44468f8f4edbd8cb4db514/roles/609ec30d147941b4a2b5508980aea7d0` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:46.150914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ae8fa5-9e97-4121-a376-638883f94a5d None admin] RBAC: Authorizing `identity:create_grant(project_id=6be022c3c7c44b9fab3d8e9fd402e509, user_id=b227b1d78f44468f8f4edbd8cb4db514, role_id=609ec30d147941b4a2b5508980aea7d0)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:46.152329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ae8fa5-9e97-4121-a376-638883f94a5d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:46.231932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 559/1123] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:54:46 2022] PUT /identity/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509/users/b227b1d78f44468f8f4edbd8cb4db514/roles/609ec30d147941b4a2b5508980aea7d0 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:54:48.073073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4d4b6a-924f-413a-914e-8f32f4c8781a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:48.073621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4d4b6a-924f-413a-914e-8f32f4c8781a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:48.073712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4d4b6a-924f-413a-914e-8f32f4c8781a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:48.074977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 565/1124] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:48.079996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc9a0d9-7016-48a3-81cd-6acdf54aa5f5 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:48.080392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc9a0d9-7016-48a3-81cd-6acdf54aa5f5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:48.080545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc9a0d9-7016-48a3-81cd-6acdf54aa5f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:48.093496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:48.099183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8cc9a0d9-7016-48a3-81cd-6acdf54aa5f5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:48.100351 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8cc9a0d9-7016-48a3-81cd-6acdf54aa5f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:48.165212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 560/1125] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:48.428838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4d4b6a-924f-413a-914e-8f32f4c8781a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:48.461111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:48.530012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4d4b6a-924f-413a-914e-8f32f4c8781a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:48.562992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab6b2c-3cc0-4e81-a704-340c620d3e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:48.563605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab6b2c-3cc0-4e81-a704-340c620d3e3e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:48.563880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab6b2c-3cc0-4e81-a704-340c620d3e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:48.564146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab6b2c-3cc0-4e81-a704-340c620d3e3e None admin] PATH_INFO: `/v3/roles/609ec30d147941b4a2b5508980aea7d0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:48.565492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ab6b2c-3cc0-4e81-a704-340c620d3e3e None admin] RBAC: Authorizing `identity:get_role(role_id=609ec30d147941b4a2b5508980aea7d0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:48.566874 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ab6b2c-3cc0-4e81-a704-340c620d3e3e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:48.568208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 566/1126] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:48 2022] GET /identity/v3/roles/609ec30d147941b4a2b5508980aea7d0 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:48.573893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc9a0d9-7016-48a3-81cd-6acdf54aa5f5 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:48.669734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc9a0d9-7016-48a3-81cd-6acdf54aa5f5 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:48.703169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363c564c-b0be-4341-ba02-d93a2c3194f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:48.703719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363c564c-b0be-4341-ba02-d93a2c3194f5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:48.703995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363c564c-b0be-4341-ba02-d93a2c3194f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:48.704264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363c564c-b0be-4341-ba02-d93a2c3194f5 None admin] PATH_INFO: `/v3/users/b227b1d78f44468f8f4edbd8cb4db514` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:48.705356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363c564c-b0be-4341-ba02-d93a2c3194f5 None admin] RBAC: Authorizing `identity:get_user(user_id=b227b1d78f44468f8f4edbd8cb4db514)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:48.706807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363c564c-b0be-4341-ba02-d93a2c3194f5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:48.708359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 561/1127] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:48 2022] GET /identity/v3/users/b227b1d78f44468f8f4edbd8cb4db514 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:48.714223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab6b2c-3cc0-4e81-a704-340c620d3e3e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:48.807585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:48.811208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab6b2c-3cc0-4e81-a704-340c620d3e3e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:48.842962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bcd6f-6c25-40e5-be89-35f146a7bcc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:48.843553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bcd6f-6c25-40e5-be89-35f146a7bcc2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:48.843832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bcd6f-6c25-40e5-be89-35f146a7bcc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:48.844104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95bcd6f-6c25-40e5-be89-35f146a7bcc2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:48.909998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95bcd6f-6c25-40e5-be89-35f146a7bcc2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:48.911997 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95bcd6f-6c25-40e5-be89-35f146a7bcc2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:49.002230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:49.006453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 567/1128] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:48 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:54:49.012360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363c564c-b0be-4341-ba02-d93a2c3194f5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:49.016205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:49.111283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363c564c-b0be-4341-ba02-d93a2c3194f5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:49.143380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92702b7a-ca98-4a4d-adfc-dd344b6e6be5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:49.143999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92702b7a-ca98-4a4d-adfc-dd344b6e6be5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:49.144281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92702b7a-ca98-4a4d-adfc-dd344b6e6be5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:49.144555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92702b7a-ca98-4a4d-adfc-dd344b6e6be5 None admin] PATH_INFO: `/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:49.146220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92702b7a-ca98-4a4d-adfc-dd344b6e6be5 None admin] RBAC: Authorizing `identity:get_project(project_id=6be022c3c7c44b9fab3d8e9fd402e509)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:49.147654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92702b7a-ca98-4a4d-adfc-dd344b6e6be5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:49.149265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 562/1129] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:54:49 2022] GET /identity/v3/projects/6be022c3c7c44b9fab3d8e9fd402e509 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:49.155351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bcd6f-6c25-40e5-be89-35f146a7bcc2 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:49.251282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95bcd6f-6c25-40e5-be89-35f146a7bcc2 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:49.283916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebd60a4-e5a0-4acd-90fb-cf805bea5466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:49.284451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd60a4-e5a0-4acd-90fb-cf805bea5466 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:49.284725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd60a4-e5a0-4acd-90fb-cf805bea5466 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:49.284996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebd60a4-e5a0-4acd-90fb-cf805bea5466 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:49.285740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebd60a4-e5a0-4acd-90fb-cf805bea5466 None admin] RBAC: Adding query filter params (role.id=609ec30d147941b4a2b5508980aea7d0, scope.project.id=6be022c3c7c44b9fab3d8e9fd402e509, user.id=b227b1d78f44468f8f4edbd8cb4db514) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:49.286581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebd60a4-e5a0-4acd-90fb-cf805bea5466 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:49.287987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebd60a4-e5a0-4acd-90fb-cf805bea5466 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:49.293495 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 568/1130] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:49 2022] GET /identity/v3/role_assignments?user.id=b227b1d78f44468f8f4edbd8cb4db514&scope.project.id=6be022c3c7c44b9fab3d8e9fd402e509&role.id=609ec30d147941b4a2b5508980aea7d0 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:51.204807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef61771-0b06-49ad-a606-97093c87e053 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:51.205356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef61771-0b06-49ad-a606-97093c87e053 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:51.205356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef61771-0b06-49ad-a606-97093c87e053 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:51.206728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 563/1131] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:51.212774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2272d8-9f87-4558-b8fa-56421e42075e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:51.213163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2272d8-9f87-4558-b8fa-56421e42075e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:51.213323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2272d8-9f87-4558-b8fa-56421e42075e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:51.231697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-db2272d8-9f87-4558-b8fa-56421e42075e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:51.232868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-db2272d8-9f87-4558-b8fa-56421e42075e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:51.287820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:51.297725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 569/1132] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:51.571251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef61771-0b06-49ad-a606-97093c87e053 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:51.640862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:51.670746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef61771-0b06-49ad-a606-97093c87e053 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:51.703851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dcdd76-a132-4eca-9735-e00b2ffbd729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:51.704473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dcdd76-a132-4eca-9735-e00b2ffbd729 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:51.704759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dcdd76-a132-4eca-9735-e00b2ffbd729 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:51.705030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dcdd76-a132-4eca-9735-e00b2ffbd729 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:51.706149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dcdd76-a132-4eca-9735-e00b2ffbd729 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:51.707574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dcdd76-a132-4eca-9735-e00b2ffbd729 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:51.709729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 564/1133] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:51.715740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2272d8-9f87-4558-b8fa-56421e42075e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:51.808123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2272d8-9f87-4558-b8fa-56421e42075e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:51.840599 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d71471-2c40-43e9-a267-a0e0c74f63aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:51.841131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d71471-2c40-43e9-a267-a0e0c74f63aa None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:51.841407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d71471-2c40-43e9-a267-a0e0c74f63aa None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:51.841699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d71471-2c40-43e9-a267-a0e0c74f63aa None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:51.842334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d71471-2c40-43e9-a267-a0e0c74f63aa None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:51.843730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d71471-2c40-43e9-a267-a0e0c74f63aa None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:51.935588 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 570/1134] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:54:51 2022] POST /identity/v3/projects => generated 311 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:54:53.821665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2f5d1-7821-44a7-82e8-4675dc148fe0 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:53.822185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2f5d1-7821-44a7-82e8-4675dc148fe0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:53.822185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba2f5d1-7821-44a7-82e8-4675dc148fe0 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:53.823519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 565/1135] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:53.828550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df90f565-a6ce-40ae-9bb7-f28bf3329b71 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:53.828977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df90f565-a6ce-40ae-9bb7-f28bf3329b71 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:53.829100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df90f565-a6ce-40ae-9bb7-f28bf3329b71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:53.847497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-df90f565-a6ce-40ae-9bb7-f28bf3329b71 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:53.848711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-df90f565-a6ce-40ae-9bb7-f28bf3329b71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:53.887214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:53.912621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 571/1136] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:54.177441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba2f5d1-7821-44a7-82e8-4675dc148fe0 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:54.277106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba2f5d1-7821-44a7-82e8-4675dc148fe0 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:54.309202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ead93e-1038-4f29-ab74-34326525dd2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:54.309838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ead93e-1038-4f29-ab74-34326525dd2b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:54.310114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ead93e-1038-4f29-ab74-34326525dd2b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:54.310378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ead93e-1038-4f29-ab74-34326525dd2b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:54.311460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ead93e-1038-4f29-ab74-34326525dd2b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:54.312876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ead93e-1038-4f29-ab74-34326525dd2b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:54.314436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 566/1137] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:54:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:54.320673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df90f565-a6ce-40ae-9bb7-f28bf3329b71 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:54.414174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df90f565-a6ce-40ae-9bb7-f28bf3329b71 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:54.447595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09459c6a-c363-41e4-9e25-5c808f71e823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:54.448122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09459c6a-c363-41e4-9e25-5c808f71e823 None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:54.448396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09459c6a-c363-41e4-9e25-5c808f71e823 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:54.448664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09459c6a-c363-41e4-9e25-5c808f71e823 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:54.449286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09459c6a-c363-41e4-9e25-5c808f71e823 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:54.450716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09459c6a-c363-41e4-9e25-5c808f71e823 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:54.574239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 572/1138] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:54:54 2022] POST /identity/v3/users => generated 286 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:54:56.484112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebaf6c-1927-4b01-b243-8747c976044c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:56.484619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebaf6c-1927-4b01-b243-8747c976044c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:56.484619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebaf6c-1927-4b01-b243-8747c976044c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:56.485949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 567/1139] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:56.491060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71118cda-9e20-4d91-971d-af47f895ca8c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:56.491469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71118cda-9e20-4d91-971d-af47f895ca8c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:56.491615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71118cda-9e20-4d91-971d-af47f895ca8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:56.509040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-71118cda-9e20-4d91-971d-af47f895ca8c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:56.510246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-71118cda-9e20-4d91-971d-af47f895ca8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:56.533201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:56.574660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 573/1140] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:54:56.842682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdebaf6c-1927-4b01-b243-8747c976044c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:56.853865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:56.941531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdebaf6c-1927-4b01-b243-8747c976044c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:56.974279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9960cc9-7d2a-46d4-ae76-1b5bb453b425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:56.974886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9960cc9-7d2a-46d4-ae76-1b5bb453b425 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:56.975169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9960cc9-7d2a-46d4-ae76-1b5bb453b425 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:56.975438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9960cc9-7d2a-46d4-ae76-1b5bb453b425 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:56.985184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9960cc9-7d2a-46d4-ae76-1b5bb453b425 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-a9960cc9-7d2a-46d4-ae76-1b5bb453b425 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:54:56.989953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:54:56.991123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:54:56.992242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 568/1141] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:54:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:54:56.996839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71118cda-9e20-4d91-971d-af47f895ca8c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:57.092919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71118cda-9e20-4d91-971d-af47f895ca8c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:57.125391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:57.126009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:57.126290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:57.126564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:57.127215 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:57.127516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:57.128862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:57.136680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 574/1142] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:54:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:57.142142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9960cc9-7d2a-46d4-ae76-1b5bb453b425 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:57.206417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:57.239420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9960cc9-7d2a-46d4-ae76-1b5bb453b425 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:57.272866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a1f75-dfd3-4871-8100-ddb183e866f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:57.273492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a1f75-dfd3-4871-8100-ddb183e866f7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:57.273803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a1f75-dfd3-4871-8100-ddb183e866f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:57.274082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a1f75-dfd3-4871-8100-ddb183e866f7 None admin] PATH_INFO: `/v3/users/7f16ee650c5746429f16e0937da1566e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:57.287428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71a1f75-dfd3-4871-8100-ddb183e866f7 None admin] RBAC: Authorizing `identity:get_user(user_id=7f16ee650c5746429f16e0937da1566e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:57.288866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71a1f75-dfd3-4871-8100-ddb183e866f7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:57.290478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 569/1143] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:54:57 2022] GET /identity/v3/users/7f16ee650c5746429f16e0937da1566e => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:57.296354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:57.321385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:57.469210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9439735d-07c7-4835-a9f8-473971e6724d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:57.501855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b70a00d-3b95-4642-9807-a15aec114e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:57.502451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70a00d-3b95-4642-9807-a15aec114e84 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:57.502730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70a00d-3b95-4642-9807-a15aec114e84 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:57.503003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b70a00d-3b95-4642-9807-a15aec114e84 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:57.567517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b70a00d-3b95-4642-9807-a15aec114e84 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:57.568933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b70a00d-3b95-4642-9807-a15aec114e84 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:57.590732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:57.663449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 575/1144] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:54:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:54:57.669322 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a1f75-dfd3-4871-8100-ddb183e866f7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:57.764579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a1f75-dfd3-4871-8100-ddb183e866f7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:57.789051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:57.797225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb50625c-484e-4b9d-b603-18f5f5ad4fca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:57.797881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb50625c-484e-4b9d-b603-18f5f5ad4fca None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:57.798161 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb50625c-484e-4b9d-b603-18f5f5ad4fca None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:57.804169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb50625c-484e-4b9d-b603-18f5f5ad4fca None admin] PATH_INFO: `/v3/projects/e7ca090827a14e3bb94291069994e44b` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:57.804169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb50625c-484e-4b9d-b603-18f5f5ad4fca None admin] RBAC: Authorizing `identity:get_project(project_id=e7ca090827a14e3bb94291069994e44b)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:57.804169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb50625c-484e-4b9d-b603-18f5f5ad4fca None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:57.804169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 570/1145] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:54:57 2022] GET /identity/v3/projects/e7ca090827a14e3bb94291069994e44b => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:57.810770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b70a00d-3b95-4642-9807-a15aec114e84 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:54:57.906706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b70a00d-3b95-4642-9807-a15aec114e84 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:54:57.927901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:54:57.938890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486aaf4a-2613-4715-b12a-faa1bafb2454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:54:57.940625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486aaf4a-2613-4715-b12a-faa1bafb2454 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:57.940625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486aaf4a-2613-4715-b12a-faa1bafb2454 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:57.940625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486aaf4a-2613-4715-b12a-faa1bafb2454 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:57.940955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486aaf4a-2613-4715-b12a-faa1bafb2454 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=e7ca090827a14e3bb94291069994e44b, user.id=7f16ee650c5746429f16e0937da1566e) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:54:57.941039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486aaf4a-2613-4715-b12a-faa1bafb2454 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:54:57.942985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486aaf4a-2613-4715-b12a-faa1bafb2454 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:54:57.948070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 576/1146] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:54:57 2022] GET /identity/v3/role_assignments?user.id=7f16ee650c5746429f16e0937da1566e&scope.project.id=e7ca090827a14e3bb94291069994e44b&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:54:59.799314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e140f26-2d7f-4da9-8b59-1b0656af4de1 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:59.799838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e140f26-2d7f-4da9-8b59-1b0656af4de1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:59.799838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e140f26-2d7f-4da9-8b59-1b0656af4de1 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:59.801262 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 571/1147] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:54:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:54:59.806419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3528d-efd1-4ee4-8944-5648c00afb35 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:54:59.806798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3528d-efd1-4ee4-8944-5648c00afb35 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:54:59.806964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3528d-efd1-4ee4-8944-5648c00afb35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:54:59.824997 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-05d3528d-efd1-4ee4-8944-5648c00afb35 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:54:59.826191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-05d3528d-efd1-4ee4-8944-5648c00afb35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:54:59.899114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 577/1148] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:54:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:00.167742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e140f26-2d7f-4da9-8b59-1b0656af4de1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:00.266938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e140f26-2d7f-4da9-8b59-1b0656af4de1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:00.298793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0144e9-d69a-4650-bb23-cc8aaaec94f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:00.299405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0144e9-d69a-4650-bb23-cc8aaaec94f2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:00.299683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0144e9-d69a-4650-bb23-cc8aaaec94f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:00.299955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0144e9-d69a-4650-bb23-cc8aaaec94f2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:00.310344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0144e9-d69a-4650-bb23-cc8aaaec94f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-8f0144e9-d69a-4650-bb23-cc8aaaec94f2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:55:00.315035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:55:00.316059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:55:00.316997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 572/1149] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:55:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:55:00.322290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3528d-efd1-4ee4-8944-5648c00afb35 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:00.417523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3528d-efd1-4ee4-8944-5648c00afb35 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:00.433270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:00.450077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:00.450675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:00.450956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:00.451231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:00.451887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:00.452190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:00.453540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:00.462149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 578/1150] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:55:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:00.467831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0144e9-d69a-4650-bb23-cc8aaaec94f2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:00.522331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:00.564144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0144e9-d69a-4650-bb23-cc8aaaec94f2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:00.596436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7ec5b4-d601-47d3-ae7d-c294bd1b1f8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:00.597044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7ec5b4-d601-47d3-ae7d-c294bd1b1f8e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:00.597324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7ec5b4-d601-47d3-ae7d-c294bd1b1f8e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:00.597612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7ec5b4-d601-47d3-ae7d-c294bd1b1f8e None admin] PATH_INFO: `/v3/users/7f16ee650c5746429f16e0937da1566e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:00.599270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7ec5b4-d601-47d3-ae7d-c294bd1b1f8e None admin] RBAC: Authorizing `identity:get_user(user_id=7f16ee650c5746429f16e0937da1566e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:00.600690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7ec5b4-d601-47d3-ae7d-c294bd1b1f8e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:00.602257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 573/1151] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:55:00 2022] GET /identity/v3/users/7f16ee650c5746429f16e0937da1566e => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:00.608032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:00.704398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0a873d-9cbf-49ba-aca5-68fca738090b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:00.736746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5636ea-2db6-4390-9737-b79c70578353 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:00.737360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5636ea-2db6-4390-9737-b79c70578353 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:00.737665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5636ea-2db6-4390-9737-b79c70578353 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:00.737943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5636ea-2db6-4390-9737-b79c70578353 None admin] PATH_INFO: `/v3/projects/e7ca090827a14e3bb94291069994e44b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:00.739100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5636ea-2db6-4390-9737-b79c70578353 None admin] RBAC: Authorizing `identity:get_project(project_id=e7ca090827a14e3bb94291069994e44b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:00.740505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5636ea-2db6-4390-9737-b79c70578353 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:00.742112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 579/1152] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:55:00 2022] GET /identity/v3/projects/e7ca090827a14e3bb94291069994e44b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:00.747737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7ec5b4-d601-47d3-ae7d-c294bd1b1f8e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:00.844182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7ec5b4-d601-47d3-ae7d-c294bd1b1f8e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:00.865597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:00.881671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978cb70b-0df4-4dd2-a094-2e49c822342d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:00.881671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cb70b-0df4-4dd2-a094-2e49c822342d None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:00.881671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cb70b-0df4-4dd2-a094-2e49c822342d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:00.881671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cb70b-0df4-4dd2-a094-2e49c822342d None admin] PATH_INFO: `/v3/projects/e7ca090827a14e3bb94291069994e44b/users/7f16ee650c5746429f16e0937da1566e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:00.885997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978cb70b-0df4-4dd2-a094-2e49c822342d None admin] RBAC: Authorizing `identity:create_grant(project_id=e7ca090827a14e3bb94291069994e44b, user_id=7f16ee650c5746429f16e0937da1566e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:00.887573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978cb70b-0df4-4dd2-a094-2e49c822342d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:01.003625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 574/1153] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:55:00 2022] PUT /identity/v3/projects/e7ca090827a14e3bb94291069994e44b/users/7f16ee650c5746429f16e0937da1566e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:55:02.856747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a097cb-6b3e-4bcb-b16a-35b19ea90034 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:02.857268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a097cb-6b3e-4bcb-b16a-35b19ea90034 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:02.857268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a097cb-6b3e-4bcb-b16a-35b19ea90034 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:02.858653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 580/1154] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:55:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:55:02.863925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d86e2b-ce6e-4aa0-90a0-769961ba0686 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:02.864318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d86e2b-ce6e-4aa0-90a0-769961ba0686 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:02.864468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d86e2b-ce6e-4aa0-90a0-769961ba0686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:02.882735 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-a0d86e2b-ce6e-4aa0-90a0-769961ba0686 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:55:02.883898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a0d86e2b-ce6e-4aa0-90a0-769961ba0686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:55:02.947579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 575/1155] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:55:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:03.223547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a097cb-6b3e-4bcb-b16a-35b19ea90034 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:03.269092 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:03.320830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a097cb-6b3e-4bcb-b16a-35b19ea90034 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:03.355267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270a70fc-a946-44b8-84c7-cef11b960c7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:03.355882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270a70fc-a946-44b8-84c7-cef11b960c7a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:03.356157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270a70fc-a946-44b8-84c7-cef11b960c7a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:03.356425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270a70fc-a946-44b8-84c7-cef11b960c7a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:03.366331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270a70fc-a946-44b8-84c7-cef11b960c7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-270a70fc-a946-44b8-84c7-cef11b960c7a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:55:03.371065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:55:03.372179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:55:03.373327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 581/1156] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:55:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:55:03.378382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d86e2b-ce6e-4aa0-90a0-769961ba0686 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:03.474893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d86e2b-ce6e-4aa0-90a0-769961ba0686 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:03.487267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:03.507714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:03.508305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:03.508582 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:03.508852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:03.509498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:03.509829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:03.511252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:03.520277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 576/1157] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:55:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:03.525756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270a70fc-a946-44b8-84c7-cef11b960c7a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:03.620681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270a70fc-a946-44b8-84c7-cef11b960c7a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:03.628235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:03.654346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86762779-75d4-42cc-b428-e96637e49c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:03.654958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86762779-75d4-42cc-b428-e96637e49c9f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:03.655231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86762779-75d4-42cc-b428-e96637e49c9f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:03.655500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86762779-75d4-42cc-b428-e96637e49c9f None admin] PATH_INFO: `/v3/users/7f16ee650c5746429f16e0937da1566e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:03.656592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86762779-75d4-42cc-b428-e96637e49c9f None admin] RBAC: Authorizing `identity:get_user(user_id=7f16ee650c5746429f16e0937da1566e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:03.658035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86762779-75d4-42cc-b428-e96637e49c9f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:03.659839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 582/1158] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:55:03 2022] GET /identity/v3/users/7f16ee650c5746429f16e0937da1566e => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:03.665402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:03.760781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16218b50-41cc-4d75-9a9f-1d79d0c76b58 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:03.792738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750b055a-7eaf-4a82-aca3-1e78693b7b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:03.793331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750b055a-7eaf-4a82-aca3-1e78693b7b89 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:03.793630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750b055a-7eaf-4a82-aca3-1e78693b7b89 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:03.793906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750b055a-7eaf-4a82-aca3-1e78693b7b89 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:03.834471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:03.858750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750b055a-7eaf-4a82-aca3-1e78693b7b89 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:03.860180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750b055a-7eaf-4a82-aca3-1e78693b7b89 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:03.954066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 577/1159] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:55:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:55:03.959876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86762779-75d4-42cc-b428-e96637e49c9f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:04.056272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86762779-75d4-42cc-b428-e96637e49c9f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:04.089436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d03234c-bc8c-4fcb-9e4b-fd044dc8abf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:04.090076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d03234c-bc8c-4fcb-9e4b-fd044dc8abf4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:04.090349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d03234c-bc8c-4fcb-9e4b-fd044dc8abf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:04.090615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d03234c-bc8c-4fcb-9e4b-fd044dc8abf4 None admin] PATH_INFO: `/v3/projects/e7ca090827a14e3bb94291069994e44b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:04.091706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d03234c-bc8c-4fcb-9e4b-fd044dc8abf4 None admin] RBAC: Authorizing `identity:get_project(project_id=e7ca090827a14e3bb94291069994e44b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:04.093111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d03234c-bc8c-4fcb-9e4b-fd044dc8abf4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:04.094740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 583/1160] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:55:03 2022] GET /identity/v3/projects/e7ca090827a14e3bb94291069994e44b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:04.100848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750b055a-7eaf-4a82-aca3-1e78693b7b89 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:04.172212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:04.197022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750b055a-7eaf-4a82-aca3-1e78693b7b89 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:04.229043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8dc31a-c940-4398-86ac-8b06d5bc8a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:04.229673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8dc31a-c940-4398-86ac-8b06d5bc8a35 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:04.229953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8dc31a-c940-4398-86ac-8b06d5bc8a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:04.230222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8dc31a-c940-4398-86ac-8b06d5bc8a35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:04.230932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8dc31a-c940-4398-86ac-8b06d5bc8a35 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=e7ca090827a14e3bb94291069994e44b, user.id=7f16ee650c5746429f16e0937da1566e) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:04.231236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8dc31a-c940-4398-86ac-8b06d5bc8a35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:04.232652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8dc31a-c940-4398-86ac-8b06d5bc8a35 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:04.238688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 578/1161] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:55:04 2022] GET /identity/v3/role_assignments?user.id=7f16ee650c5746429f16e0937da1566e&scope.project.id=e7ca090827a14e3bb94291069994e44b&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:06.129976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4d3d89-d9f2-4f97-b116-b6642a47edf4 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:06.130505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4d3d89-d9f2-4f97-b116-b6642a47edf4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:06.130505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4d3d89-d9f2-4f97-b116-b6642a47edf4 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:06.131839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 584/1162] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:55:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:55:06.136922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1c52a-3793-4407-b9d4-9b606d5ad40d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:06.137312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1c52a-3793-4407-b9d4-9b606d5ad40d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:06.137478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1c52a-3793-4407-b9d4-9b606d5ad40d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:06.155634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-adb1c52a-3793-4407-b9d4-9b606d5ad40d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:55:06.156815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-adb1c52a-3793-4407-b9d4-9b606d5ad40d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:55:06.223672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 579/1163] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:55:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:06.490522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4d3d89-d9f2-4f97-b116-b6642a47edf4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:06.533957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:06.589547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4d3d89-d9f2-4f97-b116-b6642a47edf4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:06.621935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fad966-41d5-4911-b99b-16c9266545b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:06.622520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fad966-41d5-4911-b99b-16c9266545b4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:06.622793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fad966-41d5-4911-b99b-16c9266545b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:06.623060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fad966-41d5-4911-b99b-16c9266545b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:06.686353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fad966-41d5-4911-b99b-16c9266545b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:06.687769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fad966-41d5-4911-b99b-16c9266545b4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:06.727777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:06.782044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 585/1164] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:55:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:55:06.787873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1c52a-3793-4407-b9d4-9b606d5ad40d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:06.850418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:06.883622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1c52a-3793-4407-b9d4-9b606d5ad40d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:06.915143 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a999e-6e58-4b31-88df-dc3d80cd2061 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:06.915764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a999e-6e58-4b31-88df-dc3d80cd2061 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:06.916041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a999e-6e58-4b31-88df-dc3d80cd2061 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:06.916309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a999e-6e58-4b31-88df-dc3d80cd2061 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:06.922638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3a999e-6e58-4b31-88df-dc3d80cd2061 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:06.924053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3a999e-6e58-4b31-88df-dc3d80cd2061 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-8c3a999e-6e58-4b31-88df-dc3d80cd2061 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:55:06.935470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:55:06.936667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:55:06.937754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 580/1165] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:55:06 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:55:06.943077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fad966-41d5-4911-b99b-16c9266545b4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:07.037235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fad966-41d5-4911-b99b-16c9266545b4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:07.069955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3177f243-4b10-4eea-ab6c-cbf4d72448c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:07.070537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3177f243-4b10-4eea-ab6c-cbf4d72448c2 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:07.070811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3177f243-4b10-4eea-ab6c-cbf4d72448c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:07.071077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3177f243-4b10-4eea-ab6c-cbf4d72448c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:07.071701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3177f243-4b10-4eea-ab6c-cbf4d72448c2 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:07.071998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3177f243-4b10-4eea-ab6c-cbf4d72448c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:07.073324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3177f243-4b10-4eea-ab6c-cbf4d72448c2 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:07.080121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 586/1166] 10.222.0.16 () {56 vars in 1115 bytes} [Thu Aug 25 04:55:06 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:08.940177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5298f76-002d-4d25-9fa9-64ad5db73ece None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:08.940694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5298f76-002d-4d25-9fa9-64ad5db73ece None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:08.940787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5298f76-002d-4d25-9fa9-64ad5db73ece None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:08.942616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 581/1167] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:55:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:55:08.947670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736601f-a977-4144-a661-5104d384b90e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:08.948034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736601f-a977-4144-a661-5104d384b90e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:08.948215 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736601f-a977-4144-a661-5104d384b90e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:08.961744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:08.966810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8736601f-a977-4144-a661-5104d384b90e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:55:08.967959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8736601f-a977-4144-a661-5104d384b90e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:55:09.032227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 587/1168] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:55:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:09.296288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5298f76-002d-4d25-9fa9-64ad5db73ece None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:09.395428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5298f76-002d-4d25-9fa9-64ad5db73ece None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:09.420093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:09.427874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d9096-f0c6-47fa-b6ec-4e7cba873067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:09.428456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d9096-f0c6-47fa-b6ec-4e7cba873067 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:09.428731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d9096-f0c6-47fa-b6ec-4e7cba873067 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:09.434075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d9096-f0c6-47fa-b6ec-4e7cba873067 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:09.434075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d9096-f0c6-47fa-b6ec-4e7cba873067 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:09.434075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d9096-f0c6-47fa-b6ec-4e7cba873067 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:09.557134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 582/1169] 10.222.0.16 () {60 vars in 1138 bytes} [Thu Aug 25 04:55:09 2022] POST /identity/v3/domains => generated 267 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:55:11.466032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694773bb-4f2d-49f8-a34a-9b6eafd24cd6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:11.466607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694773bb-4f2d-49f8-a34a-9b6eafd24cd6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:11.466607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694773bb-4f2d-49f8-a34a-9b6eafd24cd6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:11.467910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 588/1170] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:55:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:55:11.473884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eeabc1-2142-4aa5-9555-cca45844ff18 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:11.474312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eeabc1-2142-4aa5-9555-cca45844ff18 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:11.474419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08eeabc1-2142-4aa5-9555-cca45844ff18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:11.491917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-08eeabc1-2142-4aa5-9555-cca45844ff18 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:55:11.493110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-08eeabc1-2142-4aa5-9555-cca45844ff18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:55:11.556410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 583/1171] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:55:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:11.822609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694773bb-4f2d-49f8-a34a-9b6eafd24cd6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:11.921938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694773bb-4f2d-49f8-a34a-9b6eafd24cd6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:11.954377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac40044-92b4-42ef-88b8-613a328b738f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:11.954983 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac40044-92b4-42ef-88b8-613a328b738f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:11.955257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac40044-92b4-42ef-88b8-613a328b738f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:11.955521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac40044-92b4-42ef-88b8-613a328b738f None admin] PATH_INFO: `/v3/domains/d93d6b57848c452488c67c4f7920e037` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:11.958672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:11.966606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac40044-92b4-42ef-88b8-613a328b738f None admin] RBAC: Authorizing `identity:get_domain(domain_id=d93d6b57848c452488c67c4f7920e037)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:11.968014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac40044-92b4-42ef-88b8-613a328b738f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:11.969596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 589/1172] 10.222.0.16 () {56 vars in 1183 bytes} [Thu Aug 25 04:55:11 2022] GET /identity/v3/domains/d93d6b57848c452488c67c4f7920e037 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:11.975472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eeabc1-2142-4aa5-9555-cca45844ff18 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:12.070671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08eeabc1-2142-4aa5-9555-cca45844ff18 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:12.088080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:12.103923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9143f82e-fd74-41b2-b31b-b6139addab85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:12.104522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9143f82e-fd74-41b2-b31b-b6139addab85 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:12.104801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9143f82e-fd74-41b2-b31b-b6139addab85 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:12.105072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9143f82e-fd74-41b2-b31b-b6139addab85 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:12.105738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9143f82e-fd74-41b2-b31b-b6139addab85 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:12.107159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9143f82e-fd74-41b2-b31b-b6139addab85 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:12.220043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 584/1173] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:55:11 2022] POST /identity/v3/projects => generated 361 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:55:14.133298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85bf87d-e032-4615-9d6f-9759c2c12bde None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:14.133851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85bf87d-e032-4615-9d6f-9759c2c12bde None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:14.133851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85bf87d-e032-4615-9d6f-9759c2c12bde None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:14.135204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 590/1174] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:55:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:55:14.140291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b51258-dfb8-4780-8d15-c9800de03a5d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:14.140721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b51258-dfb8-4780-8d15-c9800de03a5d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:14.140831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b51258-dfb8-4780-8d15-c9800de03a5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:14.158694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-60b51258-dfb8-4780-8d15-c9800de03a5d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:55:14.159866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-60b51258-dfb8-4780-8d15-c9800de03a5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:55:14.223052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 585/1175] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:55:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:14.490875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85bf87d-e032-4615-9d6f-9759c2c12bde None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:14.588075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85bf87d-e032-4615-9d6f-9759c2c12bde None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:14.620123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc12b8-812d-4151-b98c-4cc78f3c5978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:14.620732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc12b8-812d-4151-b98c-4cc78f3c5978 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:14.621005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc12b8-812d-4151-b98c-4cc78f3c5978 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:14.621273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc12b8-812d-4151-b98c-4cc78f3c5978 None admin] PATH_INFO: `/v3/domains/d93d6b57848c452488c67c4f7920e037` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:14.622387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc12b8-812d-4151-b98c-4cc78f3c5978 None admin] RBAC: Authorizing `identity:get_domain(domain_id=d93d6b57848c452488c67c4f7920e037)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:14.623789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc12b8-812d-4151-b98c-4cc78f3c5978 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:14.625315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 591/1176] 10.222.0.16 () {56 vars in 1183 bytes} [Thu Aug 25 04:55:14 2022] GET /identity/v3/domains/d93d6b57848c452488c67c4f7920e037 => generated 267 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:14.631802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b51258-dfb8-4780-8d15-c9800de03a5d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:14.724268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:14.727105 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b51258-dfb8-4780-8d15-c9800de03a5d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:14.759907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a96daf-31d3-4799-9d48-161ae927366d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:14.760440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a96daf-31d3-4799-9d48-161ae927366d None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:14.760717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a96daf-31d3-4799-9d48-161ae927366d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:14.760997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a96daf-31d3-4799-9d48-161ae927366d None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:14.761940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a96daf-31d3-4799-9d48-161ae927366d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:14.763598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a96daf-31d3-4799-9d48-161ae927366d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:14.853819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 586/1177] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:55:14 2022] POST /identity/v3/users => generated 311 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:55:16.757303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d66d91-f37a-4be0-b3f6-abc0cace505a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:16.757888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d66d91-f37a-4be0-b3f6-abc0cace505a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:16.757888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d66d91-f37a-4be0-b3f6-abc0cace505a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:16.759221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 592/1178] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:55:16.764674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d95c2a-41a2-4056-b3d1-1832b4122da5 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:16.765085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d95c2a-41a2-4056-b3d1-1832b4122da5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:16.765227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d95c2a-41a2-4056-b3d1-1832b4122da5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:16.782734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-19d95c2a-41a2-4056-b3d1-1832b4122da5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:55:16.783909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-19d95c2a-41a2-4056-b3d1-1832b4122da5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:55:16.848007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 587/1179] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:55:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:17.115412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d66d91-f37a-4be0-b3f6-abc0cace505a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:17.176928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:17.213726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d66d91-f37a-4be0-b3f6-abc0cace505a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:17.246757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a57742-9143-40d9-a395-fe98446ae2eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:17.247384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a57742-9143-40d9-a395-fe98446ae2eb None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:17.247663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a57742-9143-40d9-a395-fe98446ae2eb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:17.247933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a57742-9143-40d9-a395-fe98446ae2eb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:17.257777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a57742-9143-40d9-a395-fe98446ae2eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-06a57742-9143-40d9-a395-fe98446ae2eb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:55:17.262797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:55:17.263963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:55:17.264907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 593/1180] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:55:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:55:17.269957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d95c2a-41a2-4056-b3d1-1832b4122da5 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:17.350142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:17.364558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d95c2a-41a2-4056-b3d1-1832b4122da5 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:17.397284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:17.397910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:17.398185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:17.398454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:17.399103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:17.399400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:17.400848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:17.408627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 588/1181] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:55:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:17.420003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a57742-9143-40d9-a395-fe98446ae2eb None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:17.511502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a57742-9143-40d9-a395-fe98446ae2eb None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:17.534180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:17.544820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bc50bb-6d10-43a0-8edf-313d5cabd77c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:17.545433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bc50bb-6d10-43a0-8edf-313d5cabd77c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:17.545741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bc50bb-6d10-43a0-8edf-313d5cabd77c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:17.546023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25bc50bb-6d10-43a0-8edf-313d5cabd77c None admin] PATH_INFO: `/v3/users/529c7f00666140f28fce1ac7e5907a3e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:17.558569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bc50bb-6d10-43a0-8edf-313d5cabd77c None admin] RBAC: Authorizing `identity:get_user(user_id=529c7f00666140f28fce1ac7e5907a3e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:17.560001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25bc50bb-6d10-43a0-8edf-313d5cabd77c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:17.561562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 594/1182] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:55:17 2022] GET /identity/v3/users/529c7f00666140f28fce1ac7e5907a3e => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:17.567476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:17.664413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29778f1-bf9d-4603-8e3c-596df8fbd77e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:17.697103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a41dfb-6b55-40ec-901c-60e0209ed36e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:17.697728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a41dfb-6b55-40ec-901c-60e0209ed36e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:17.698010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a41dfb-6b55-40ec-901c-60e0209ed36e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:17.698284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a41dfb-6b55-40ec-901c-60e0209ed36e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:17.715151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:17.763707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a41dfb-6b55-40ec-901c-60e0209ed36e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:17.765148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a41dfb-6b55-40ec-901c-60e0209ed36e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:17.858579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 589/1183] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:55:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:55:17.865136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bc50bb-6d10-43a0-8edf-313d5cabd77c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:17.962063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25bc50bb-6d10-43a0-8edf-313d5cabd77c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:17.995425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7cad7c8-e10f-4221-a623-dc1a24a6427e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:17.996043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cad7c8-e10f-4221-a623-dc1a24a6427e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:17.996321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cad7c8-e10f-4221-a623-dc1a24a6427e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:17.996590 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7cad7c8-e10f-4221-a623-dc1a24a6427e None admin] PATH_INFO: `/v3/projects/721e52a88e0f4cc18c6b3a277699b188` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:17.997719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cad7c8-e10f-4221-a623-dc1a24a6427e None admin] RBAC: Authorizing `identity:get_project(project_id=721e52a88e0f4cc18c6b3a277699b188)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:17.999123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7cad7c8-e10f-4221-a623-dc1a24a6427e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:18.000709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 595/1184] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:55:17 2022] GET /identity/v3/projects/721e52a88e0f4cc18c6b3a277699b188 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:18.006663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a41dfb-6b55-40ec-901c-60e0209ed36e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:18.053998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:18.103188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a41dfb-6b55-40ec-901c-60e0209ed36e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:18.134919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620bd82d-b22a-47bc-9012-8f8658a78a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:18.135504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bd82d-b22a-47bc-9012-8f8658a78a4a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:18.135780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bd82d-b22a-47bc-9012-8f8658a78a4a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:18.136055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bd82d-b22a-47bc-9012-8f8658a78a4a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:18.136758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620bd82d-b22a-47bc-9012-8f8658a78a4a None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=721e52a88e0f4cc18c6b3a277699b188, user.id=529c7f00666140f28fce1ac7e5907a3e) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:18.137060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620bd82d-b22a-47bc-9012-8f8658a78a4a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:18.138505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620bd82d-b22a-47bc-9012-8f8658a78a4a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:18.143600 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 590/1185] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:55:18 2022] GET /identity/v3/role_assignments?user.id=529c7f00666140f28fce1ac7e5907a3e&scope.project.id=721e52a88e0f4cc18c6b3a277699b188&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:19.993918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a483d8b3-a247-4eff-b200-314ec12a874a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:19.994461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a483d8b3-a247-4eff-b200-314ec12a874a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:19.994461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a483d8b3-a247-4eff-b200-314ec12a874a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:19.995803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 596/1186] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:55:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:55:20.000843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe993c3b-4319-4c38-a6d8-b6b6cb24949d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:20.001205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe993c3b-4319-4c38-a6d8-b6b6cb24949d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:20.001391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe993c3b-4319-4c38-a6d8-b6b6cb24949d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:20.019450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-fe993c3b-4319-4c38-a6d8-b6b6cb24949d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:55:20.020609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-fe993c3b-4319-4c38-a6d8-b6b6cb24949d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:55:20.083710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 591/1187] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:55:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:20.349138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a483d8b3-a247-4eff-b200-314ec12a874a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:20.399516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:20.447128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a483d8b3-a247-4eff-b200-314ec12a874a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:20.479625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d63d2-3a8d-4f6f-959e-8e899b9f73a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:20.480232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d63d2-3a8d-4f6f-959e-8e899b9f73a3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:20.480506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d63d2-3a8d-4f6f-959e-8e899b9f73a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:20.480773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d63d2-3a8d-4f6f-959e-8e899b9f73a3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:20.491178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784d63d2-3a8d-4f6f-959e-8e899b9f73a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-784d63d2-3a8d-4f6f-959e-8e899b9f73a3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:55:20.495908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:55:20.496993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:55:20.498017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 597/1188] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:55:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:55:20.502822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe993c3b-4319-4c38-a6d8-b6b6cb24949d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:20.544373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:20.599446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe993c3b-4319-4c38-a6d8-b6b6cb24949d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:20.631483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:20.632096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:20.632378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:20.632650 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:20.633302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:20.633626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:20.634988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:20.642686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 592/1189] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:55:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:20.648868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d63d2-3a8d-4f6f-959e-8e899b9f73a3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:20.749512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d63d2-3a8d-4f6f-959e-8e899b9f73a3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:20.760221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:20.783244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb351932-5c6b-4867-83da-9a475f847c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:20.783994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb351932-5c6b-4867-83da-9a475f847c53 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:20.784321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb351932-5c6b-4867-83da-9a475f847c53 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:20.784607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb351932-5c6b-4867-83da-9a475f847c53 None admin] PATH_INFO: `/v3/users/529c7f00666140f28fce1ac7e5907a3e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:20.785749 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb351932-5c6b-4867-83da-9a475f847c53 None admin] RBAC: Authorizing `identity:get_user(user_id=529c7f00666140f28fce1ac7e5907a3e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:20.787172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb351932-5c6b-4867-83da-9a475f847c53 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:20.788732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 598/1190] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:55:20 2022] GET /identity/v3/users/529c7f00666140f28fce1ac7e5907a3e => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:20.794499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:20.890632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0228511a-bbb0-481a-8751-88d4a3d7ba3f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:20.892859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:20.924591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9206b1-ff8a-44f6-aef2-4bc91ee65e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:20.925235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9206b1-ff8a-44f6-aef2-4bc91ee65e8b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:20.925518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9206b1-ff8a-44f6-aef2-4bc91ee65e8b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:20.925821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9206b1-ff8a-44f6-aef2-4bc91ee65e8b None admin] PATH_INFO: `/v3/projects/721e52a88e0f4cc18c6b3a277699b188` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:20.926937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9206b1-ff8a-44f6-aef2-4bc91ee65e8b None admin] RBAC: Authorizing `identity:get_project(project_id=721e52a88e0f4cc18c6b3a277699b188)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:20.928364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9206b1-ff8a-44f6-aef2-4bc91ee65e8b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:20.929990 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 593/1191] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:55:20 2022] GET /identity/v3/projects/721e52a88e0f4cc18c6b3a277699b188 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:20.935700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb351932-5c6b-4867-83da-9a475f847c53 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:21.032881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb351932-5c6b-4867-83da-9a475f847c53 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:21.064598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8a41eb-d220-4710-a81d-14a47bee1c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:21.065235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8a41eb-d220-4710-a81d-14a47bee1c54 None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:21.066079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8a41eb-d220-4710-a81d-14a47bee1c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:21.066356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8a41eb-d220-4710-a81d-14a47bee1c54 None admin] PATH_INFO: `/v3/projects/721e52a88e0f4cc18c6b3a277699b188/users/529c7f00666140f28fce1ac7e5907a3e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:21.068241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8a41eb-d220-4710-a81d-14a47bee1c54 None admin] RBAC: Authorizing `identity:create_grant(project_id=721e52a88e0f4cc18c6b3a277699b188, user_id=529c7f00666140f28fce1ac7e5907a3e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:21.069690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8a41eb-d220-4710-a81d-14a47bee1c54 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:21.172880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 599/1192] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:55:20 2022] PUT /identity/v3/projects/721e52a88e0f4cc18c6b3a277699b188/users/529c7f00666140f28fce1ac7e5907a3e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:55:23.038083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72205a3-4e3f-4fcf-95d6-b74a351e5031 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:23.038608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72205a3-4e3f-4fcf-95d6-b74a351e5031 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:23.038608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72205a3-4e3f-4fcf-95d6-b74a351e5031 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:23.039943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 594/1193] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:55:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:55:23.045117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea925f23-a3fc-4d7d-bf39-8cc01a76104d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:23.045534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea925f23-a3fc-4d7d-bf39-8cc01a76104d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:23.045693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea925f23-a3fc-4d7d-bf39-8cc01a76104d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:23.064077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ea925f23-a3fc-4d7d-bf39-8cc01a76104d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:55:23.065242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ea925f23-a3fc-4d7d-bf39-8cc01a76104d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:55:23.093448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:23.129494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 600/1194] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:55:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:55:23.397429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72205a3-4e3f-4fcf-95d6-b74a351e5031 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:23.494726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72205a3-4e3f-4fcf-95d6-b74a351e5031 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:23.526789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e5357-a7da-4926-b119-56385fb4040b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:23.527396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e5357-a7da-4926-b119-56385fb4040b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:23.527672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e5357-a7da-4926-b119-56385fb4040b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:23.527937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e5357-a7da-4926-b119-56385fb4040b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:23.537574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8e5357-a7da-4926-b119-56385fb4040b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-dd8e5357-a7da-4926-b119-56385fb4040b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:55:23.542251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:55:23.543388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:55:23.544453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 595/1195] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:55:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:55:23.549066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea925f23-a3fc-4d7d-bf39-8cc01a76104d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:23.644050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea925f23-a3fc-4d7d-bf39-8cc01a76104d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:23.677431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:23.677994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:23.678269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:23.678538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:23.679180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:23.679481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:23.680830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:23.688607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 601/1196] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:55:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:23.695164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e5357-a7da-4926-b119-56385fb4040b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:23.725633 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:23.792401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e5357-a7da-4926-b119-56385fb4040b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:23.825512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91348df5-d24f-411c-94cf-f20cc12490c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:23.826150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91348df5-d24f-411c-94cf-f20cc12490c1 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:23.826426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91348df5-d24f-411c-94cf-f20cc12490c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:23.826695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91348df5-d24f-411c-94cf-f20cc12490c1 None admin] PATH_INFO: `/v3/users/529c7f00666140f28fce1ac7e5907a3e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:23.827792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91348df5-d24f-411c-94cf-f20cc12490c1 None admin] RBAC: Authorizing `identity:get_user(user_id=529c7f00666140f28fce1ac7e5907a3e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:23.829213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91348df5-d24f-411c-94cf-f20cc12490c1 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:23.830960 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 596/1197] 10.222.0.16 () {56 vars in 1177 bytes} [Thu Aug 25 04:55:23 2022] GET /identity/v3/users/529c7f00666140f28fce1ac7e5907a3e => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:23.837055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:23.867525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:23.934229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836dbfad-f552-469a-8258-e6d7ba62aae3 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:23.966887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5cef3a-a26e-455c-8b98-d66323769244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:23.967474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5cef3a-a26e-455c-8b98-d66323769244 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:23.967748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5cef3a-a26e-455c-8b98-d66323769244 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:23.968016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5cef3a-a26e-455c-8b98-d66323769244 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:24.030940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5cef3a-a26e-455c-8b98-d66323769244 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:24.032347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5cef3a-a26e-455c-8b98-d66323769244 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:24.060391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:24.126255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 602/1198] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:55:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:55:24.132216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91348df5-d24f-411c-94cf-f20cc12490c1 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:24.224765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:24.227658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91348df5-d24f-411c-94cf-f20cc12490c1 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:24.259605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c14f0c-dd97-4539-94da-c3777a5236fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:24.260222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c14f0c-dd97-4539-94da-c3777a5236fc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:24.260499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c14f0c-dd97-4539-94da-c3777a5236fc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:24.260769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c14f0c-dd97-4539-94da-c3777a5236fc None admin] PATH_INFO: `/v3/projects/721e52a88e0f4cc18c6b3a277699b188` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:24.261889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c14f0c-dd97-4539-94da-c3777a5236fc None admin] RBAC: Authorizing `identity:get_project(project_id=721e52a88e0f4cc18c6b3a277699b188)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:24.263325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c14f0c-dd97-4539-94da-c3777a5236fc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:24.265148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 597/1199] 10.222.0.16 () {56 vars in 1186 bytes} [Thu Aug 25 04:55:24 2022] GET /identity/v3/projects/721e52a88e0f4cc18c6b3a277699b188 => generated 361 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:24.271733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5cef3a-a26e-455c-8b98-d66323769244 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:55:24.366638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5cef3a-a26e-455c-8b98-d66323769244 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:55:24.393146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:24.399269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3912d94-8199-4c9e-bf65-43742338b50a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:55:24.399859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3912d94-8199-4c9e-bf65-43742338b50a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:55:24.400138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3912d94-8199-4c9e-bf65-43742338b50a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:55:24.400408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3912d94-8199-4c9e-bf65-43742338b50a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:55:24.401119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3912d94-8199-4c9e-bf65-43742338b50a None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=721e52a88e0f4cc18c6b3a277699b188, user.id=529c7f00666140f28fce1ac7e5907a3e) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:55:24.401426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3912d94-8199-4c9e-bf65-43742338b50a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:55:24.402872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3912d94-8199-4c9e-bf65-43742338b50a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:55:24.408315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 603/1200] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:55:24 2022] GET /identity/v3/role_assignments?user.id=529c7f00666140f28fce1ac7e5907a3e&scope.project.id=721e52a88e0f4cc18c6b3a277699b188&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:55:34.267059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=16 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:55:34.447660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:40.471245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04ff73-4e92-4ba4-8da0-62981d031773 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:40.471821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04ff73-4e92-4ba4-8da0-62981d031773 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:40.471821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c04ff73-4e92-4ba4-8da0-62981d031773 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:40.473184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 598/1201] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:56:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 04:56:40.478427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e0dca-e831-4e4f-b1e8-bc572c72022e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:40.478852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e0dca-e831-4e4f-b1e8-bc572c72022e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:40.478979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e0dca-e831-4e4f-b1e8-bc572c72022e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:40.496615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-937e0dca-e831-4e4f-b1e8-bc572c72022e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:56:40.501904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-937e0dca-e831-4e4f-b1e8-bc572c72022e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:56:40.563355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 604/1202] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:56:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:56:40.828460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c04ff73-4e92-4ba4-8da0-62981d031773 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:40.928514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c04ff73-4e92-4ba4-8da0-62981d031773 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:40.962090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271fa228-d841-493e-8983-aeb12210823a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:40.962711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fa228-d841-493e-8983-aeb12210823a None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:40.962988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fa228-d841-493e-8983-aeb12210823a None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:40.963259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fa228-d841-493e-8983-aeb12210823a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:40.973551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271fa228-d841-493e-8983-aeb12210823a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:40.975023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271fa228-d841-493e-8983-aeb12210823a None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-271fa228-d841-493e-8983-aeb12210823a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:40.989086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:40.990308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:40.991466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:40.991466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:40.991466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:40.991466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 599/1203] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:40.996455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e0dca-e831-4e4f-b1e8-bc572c72022e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:41.093008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e0dca-e831-4e4f-b1e8-bc572c72022e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:41.120458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:41.126564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1587dd8-6432-41fc-be28-2c6a51b23b1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:41.127159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1587dd8-6432-41fc-be28-2c6a51b23b1a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:41.127434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1587dd8-6432-41fc-be28-2c6a51b23b1a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:41.127701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1587dd8-6432-41fc-be28-2c6a51b23b1a None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:41.128336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1587dd8-6432-41fc-be28-2c6a51b23b1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:41.128642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1587dd8-6432-41fc-be28-2c6a51b23b1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:41.130044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1587dd8-6432-41fc-be28-2c6a51b23b1a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:41.141567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 605/1204] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:41.147737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271fa228-d841-493e-8983-aeb12210823a None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:41.199855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:41.245793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271fa228-d841-493e-8983-aeb12210823a None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:41.279893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb414dad-1506-4c2c-bc69-0fa63be387a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:41.280486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb414dad-1506-4c2c-bc69-0fa63be387a0 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:41.280762 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb414dad-1506-4c2c-bc69-0fa63be387a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:41.281031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb414dad-1506-4c2c-bc69-0fa63be387a0 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:41.281682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb414dad-1506-4c2c-bc69-0fa63be387a0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:41.283122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb414dad-1506-4c2c-bc69-0fa63be387a0 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:41.450152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 600/1205] 10.222.0.16 () {60 vars in 1132 bytes} [Thu Aug 25 04:56:41 2022] POST /identity/v3/users => generated 251 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:56:43.355830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342790f8-f3cc-4780-9552-5e7a0cf1db60 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:43.356398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342790f8-f3cc-4780-9552-5e7a0cf1db60 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:43.356398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342790f8-f3cc-4780-9552-5e7a0cf1db60 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:43.358743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 606/1206] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:56:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:56:43.363858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd21d1a4-d962-46af-980b-dcedb1a07f64 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:43.364248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd21d1a4-d962-46af-980b-dcedb1a07f64 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:43.364410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd21d1a4-d962-46af-980b-dcedb1a07f64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:43.382144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-cd21d1a4-d962-46af-980b-dcedb1a07f64 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:56:43.383335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cd21d1a4-d962-46af-980b-dcedb1a07f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:56:43.448383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 601/1207] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:56:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:56:43.714508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342790f8-f3cc-4780-9552-5e7a0cf1db60 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:43.815515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342790f8-f3cc-4780-9552-5e7a0cf1db60 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:43.849354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389d708c-5515-4395-b577-c1ac8e53afa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:43.850003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d708c-5515-4395-b577-c1ac8e53afa7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:43.850286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d708c-5515-4395-b577-c1ac8e53afa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:43.850564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d708c-5515-4395-b577-c1ac8e53afa7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:43.860394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389d708c-5515-4395-b577-c1ac8e53afa7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-389d708c-5515-4395-b577-c1ac8e53afa7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:56:43.865048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:43.866461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:43.867417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 607/1208] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:56:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:43.872158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd21d1a4-d962-46af-980b-dcedb1a07f64 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:43.912241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:43.973059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd21d1a4-d962-46af-980b-dcedb1a07f64 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:44.006440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:44.007057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:44.007344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:44.007614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:44.008276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:44.008580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:44.009986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:44.021740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 602/1209] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:56:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:44.024838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389d708c-5515-4395-b577-c1ac8e53afa7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:44.076876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:44.131940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389d708c-5515-4395-b577-c1ac8e53afa7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:44.166302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:44.166938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:44.167218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:44.167493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:44.178669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:44.186626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:44.194975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:44.196201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:44.197163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:44.197163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:44.197163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:44.197163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 608/1210] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:44.202802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:44.299525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:44.302543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb85c6d-5615-42ff-ae9e-626880a6d6b8 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:44.336868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:44.343202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:44.343202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:44.343202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:44.343202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:44.343202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:44.343202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:44.351743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 603/1211] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:44.357977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:44.456598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b14109-1aac-4f57-8171-591cd232209d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:44.458169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:44.490097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:44.490706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:44.490980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:44.491246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:44.498108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:44.499528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:56:44.510868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:44.514884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:44.516079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 609/1212] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:44.518560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:44.617192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574394bd-915e-4ede-ba01-bdebe89269fc None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:44.649439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:44.655328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:44.655328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:44.655328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:44.655328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:44.655328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:44.655328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:44.669962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 604/1213] 10.222.0.16 () {56 vars in 1143 bytes} [Thu Aug 25 04:56:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:44.675773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:44.772241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c4d660-912f-4e29-988b-34ab93685191 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:44.803562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db6e5bf-121a-4604-8a51-de1975cccb9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:44.804146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db6e5bf-121a-4604-8a51-de1975cccb9b None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:44.804426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db6e5bf-121a-4604-8a51-de1975cccb9b None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:44.804692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db6e5bf-121a-4604-8a51-de1975cccb9b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:44.833478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:44.869665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db6e5bf-121a-4604-8a51-de1975cccb9b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:44.871022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db6e5bf-121a-4604-8a51-de1975cccb9b None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:44.966732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 610/1214] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:56:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 25 04:56:44.972627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:44.991028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:45.071342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568d6279-7f0d-4eff-93f5-f23c6e9b04e9 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:45.104671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:45.105285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:45.105563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:45.105862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:45.116105 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:45.118362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:45.132497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:45.133626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:45.134798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:45.134798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:45.134798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:45.134798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 605/1215] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:45.139435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db6e5bf-121a-4604-8a51-de1975cccb9b None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:45.199119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:45.238917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db6e5bf-121a-4604-8a51-de1975cccb9b None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:45.271171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:45.271773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:45.272142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:45.272472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:45.273102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:45.273400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:45.274784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:45.286147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 611/1216] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:45.291888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:45.361076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:45.391971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8a7d04-5061-477c-b714-78230f9eac4b None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:45.424104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:45.424719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:45.425000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:45.425271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:45.431717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:45.433259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:56:45.444115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:45.445181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:45.446618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 606/1217] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:56:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:45.451605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:45.550703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190fb9af-d9ee-496a-b635-2f3682281f9a None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:45.573295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:45.585663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f441a6e-cd17-437b-a1ce-be2425df252e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:45.586331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f441a6e-cd17-437b-a1ce-be2425df252e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:45.586618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f441a6e-cd17-437b-a1ce-be2425df252e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:45.586896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f441a6e-cd17-437b-a1ce-be2425df252e None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:45.587587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f441a6e-cd17-437b-a1ce-be2425df252e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:45.587895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f441a6e-cd17-437b-a1ce-be2425df252e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:45.589536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f441a6e-cd17-437b-a1ce-be2425df252e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:45.603945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 612/1218] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:56:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 04:56:45.610126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:45.707556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba524ce5-0108-45e1-9ef9-6a2d289c6c93 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:45.734352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:45.739946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae780d-3072-4e84-afcd-bd101d4c2a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:45.740534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae780d-3072-4e84-afcd-bd101d4c2a72 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:45.740815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae780d-3072-4e84-afcd-bd101d4c2a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:45.741085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae780d-3072-4e84-afcd-bd101d4c2a72 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:45.741839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae780d-3072-4e84-afcd-bd101d4c2a72 None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=e1ffa92d7fb546399bc404053c201426) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:45.742149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae780d-3072-4e84-afcd-bd101d4c2a72 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:45.743562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ae780d-3072-4e84-afcd-bd101d4c2a72 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:45.749191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 607/1219] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:56:45 2022] GET /identity/v3/role_assignments?user.id=e1ffa92d7fb546399bc404053c201426&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:47.636298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eea7c4-a852-4eeb-aaaf-ae252b8e68a4 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:47.636806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eea7c4-a852-4eeb-aaaf-ae252b8e68a4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:47.636900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eea7c4-a852-4eeb-aaaf-ae252b8e68a4 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:47.638223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 613/1220] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:56:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:56:47.643872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6203e509-a274-4464-959d-4de0827555c1 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:47.644292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6203e509-a274-4464-959d-4de0827555c1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:47.644461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6203e509-a274-4464-959d-4de0827555c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:47.663087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6203e509-a274-4464-959d-4de0827555c1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:56:47.664298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6203e509-a274-4464-959d-4de0827555c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:56:47.729088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 608/1221] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:56:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:56:47.996508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7eea7c4-a852-4eeb-aaaf-ae252b8e68a4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:48.096513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7eea7c4-a852-4eeb-aaaf-ae252b8e68a4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:48.131189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebc79cc-2df9-4e83-84e9-d28a8b1fbc31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:48.131807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebc79cc-2df9-4e83-84e9-d28a8b1fbc31 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:48.132083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebc79cc-2df9-4e83-84e9-d28a8b1fbc31 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:48.132350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebc79cc-2df9-4e83-84e9-d28a8b1fbc31 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:48.142325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebc79cc-2df9-4e83-84e9-d28a8b1fbc31 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-bebc79cc-2df9-4e83-84e9-d28a8b1fbc31 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:56:48.146952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:48.148018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:48.148891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 614/1222] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:56:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:48.153982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6203e509-a274-4464-959d-4de0827555c1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:48.255931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6203e509-a274-4464-959d-4de0827555c1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:48.278195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:48.288399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:48.289006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:48.289287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:48.289733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:48.290396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:48.290701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:48.292062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:48.300244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 609/1223] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:56:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:48.306136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebc79cc-2df9-4e83-84e9-d28a8b1fbc31 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:48.349727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:48.404996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebc79cc-2df9-4e83-84e9-d28a8b1fbc31 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:48.437512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:48.438161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:48.438440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:48.438715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:48.448281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:48.449755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:48.464516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:48.465638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:48.474538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:48.474538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:48.474538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:48.474538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 615/1224] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:48.474825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:48.569412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030fb5c4-df5f-4359-97de-4fa656faaba4 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:48.602558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:48.603209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:48.609345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:48.609345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:48.609345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:48.609345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:48.609345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:48.617496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 610/1225] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:48.623737 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:48.715955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:48.722565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795553b5-2d65-4dde-b2c2-75716d7abbd9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:48.754738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:48.755347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:48.755621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:48.755888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:48.762540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:48.764586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:56:48.775779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:48.776916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:48.778256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 616/1226] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:48 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:48.782654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:48.829657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:48.881379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39685a20-a59f-40c8-8d05-96ecc8e38d8e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:48.914374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:48.914978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:48.915259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:48.915534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:48.916254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:48.916563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:48.918233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:48.934547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 611/1227] 10.222.0.16 () {56 vars in 1143 bytes} [Thu Aug 25 04:56:48 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:48.940456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:49.039101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7955a4b1-5a20-4dd1-8511-5c5c6382a980 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:49.071440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:49.072056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:49.072332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:49.072601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:49.083268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:49.084655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:49.099239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:49.101148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:49.102303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:49.102303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:49.102303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:49.102303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 617/1228] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:49.106336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:49.204962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1cd07d-edbc-4b1f-b4a6-8b7ddd423c4f None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:49.207258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:49.238996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:49.239606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:49.239885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:49.240157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:49.240788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:49.241089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:49.242484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:49.254020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 612/1229] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:49.259808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:49.270654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:49.360582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a684ba-eeb9-41dc-beb7-6ac4dce1a50d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:49.393840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:49.394562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:49.394838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:49.395106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:49.401424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:49.402877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:56:49.413781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:49.414892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:49.416018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 618/1230] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:56:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:49.420943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:49.518922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d33232-a492-46c3-9cff-ac445e29b890 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:49.550433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220acac2-5838-4183-b604-08f477ad5d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:49.551039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220acac2-5838-4183-b604-08f477ad5d54 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:49.551318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220acac2-5838-4183-b604-08f477ad5d54 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:49.551588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220acac2-5838-4183-b604-08f477ad5d54 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:49.552276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220acac2-5838-4183-b604-08f477ad5d54 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:49.552580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220acac2-5838-4183-b604-08f477ad5d54 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:49.554027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220acac2-5838-4183-b604-08f477ad5d54 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:49.566051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 613/1231] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:56:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:49.572270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:49.637625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:49.671744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2ad99c-5f72-4054-aa82-18207cc024ce None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:49.703715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1370f4b5-7245-470c-8217-77d189ecb01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:49.704361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370f4b5-7245-470c-8217-77d189ecb01c None admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:49.704638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370f4b5-7245-470c-8217-77d189ecb01c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:49.704911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370f4b5-7245-470c-8217-77d189ecb01c None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/e1ffa92d7fb546399bc404053c201426/roles/c9f7a48328df4a7ea7a5981959f395f5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:49.719295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1370f4b5-7245-470c-8217-77d189ecb01c None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=e1ffa92d7fb546399bc404053c201426, role_id=c9f7a48328df4a7ea7a5981959f395f5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:49.720718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1370f4b5-7245-470c-8217-77d189ecb01c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:49.810067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 619/1232] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:56:49 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/e1ffa92d7fb546399bc404053c201426/roles/c9f7a48328df4a7ea7a5981959f395f5 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:56:51.657100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c623a92-1481-44ee-bc78-9dd4e16dc287 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:51.657659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c623a92-1481-44ee-bc78-9dd4e16dc287 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:51.657775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c623a92-1481-44ee-bc78-9dd4e16dc287 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:51.659072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 614/1233] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:56:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:56:51.664135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b82e65-3081-462d-b77f-e7f153c0c139 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:51.664524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b82e65-3081-462d-b77f-e7f153c0c139 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:51.664683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b82e65-3081-462d-b77f-e7f153c0c139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:51.682158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-25b82e65-3081-462d-b77f-e7f153c0c139 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:56:51.687930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-25b82e65-3081-462d-b77f-e7f153c0c139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:56:51.748703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 620/1234] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:56:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:56:52.015434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c623a92-1481-44ee-bc78-9dd4e16dc287 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:52.041002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:52.118682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c623a92-1481-44ee-bc78-9dd4e16dc287 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:52.152658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57728f8-7e1c-4d0b-af96-67b478405fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:52.153277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57728f8-7e1c-4d0b-af96-67b478405fe1 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:52.153555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57728f8-7e1c-4d0b-af96-67b478405fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:52.154460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57728f8-7e1c-4d0b-af96-67b478405fe1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:52.164350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57728f8-7e1c-4d0b-af96-67b478405fe1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-e57728f8-7e1c-4d0b-af96-67b478405fe1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:56:52.169087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:52.171385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:52.172467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 615/1235] 10.222.0.16 () {56 vars in 1102 bytes} [Thu Aug 25 04:56:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:52.176240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b82e65-3081-462d-b77f-e7f153c0c139 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:52.222242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:52.273334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b82e65-3081-462d-b77f-e7f153c0c139 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:52.305031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:52.307415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:52.307415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:52.307415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:52.307415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] RBAC: Adding query filter params (name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:52.307415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:52.308499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:52.317516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 621/1236] 10.222.0.16 () {56 vars in 1103 bytes} [Thu Aug 25 04:56:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:52.323257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57728f8-7e1c-4d0b-af96-67b478405fe1 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:52.406878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:52.421991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57728f8-7e1c-4d0b-af96-67b478405fe1 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:52.455416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:52.456036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:52.456316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:52.456596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:52.467032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:52.468622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:52.482902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:52.483885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:52.484871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:52.484871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:52.484871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:52.484871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 616/1237] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:52.490295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:52.588807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af049e7-21c2-4499-8bb5-08aaeef5f108 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:52.597525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:52.621395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:52.622013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:52.622291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:52.623580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:52.624639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:52.624939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:52.626535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:52.638337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 622/1238] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:52.644047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:52.742911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5375fb4b-42fe-443e-acb8-a9ab3072bf25 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:52.775854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:52.776473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:52.776750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:52.777020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:52.779765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:52.784322 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:52.785790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:56:52.797297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:52.798471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:52.799641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 617/1239] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:52 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:52.804192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:52.901977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6cbd82-606e-4a1d-a0bc-364953d64a42 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:52.935808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:52.936411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:52.936693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:52.936965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:52.937718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:52.938028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:52.939457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:52.956035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 623/1240] 10.222.0.16 () {56 vars in 1143 bytes} [Thu Aug 25 04:56:52 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:52.962411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:53.061895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4747f49-ae94-42df-98a8-ca11b54976d1 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:53.095401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4480f998-ec30-49dd-b5a8-ffa264988e5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:53.099205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4480f998-ec30-49dd-b5a8-ffa264988e5c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:53.099205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4480f998-ec30-49dd-b5a8-ffa264988e5c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:53.099205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4480f998-ec30-49dd-b5a8-ffa264988e5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:53.155826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:53.162216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4480f998-ec30-49dd-b5a8-ffa264988e5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:53.163666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4480f998-ec30-49dd-b5a8-ffa264988e5c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:53.258876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 618/1241] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:56:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:56:53.264711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:53.292206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:53.361496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6100c9-cc3a-468b-902b-7d525a72fb25 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:53.394722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:53.395340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:53.395615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:53.395879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:53.406291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:53.407646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:53.421650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:53.422704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:53.423633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:53.423633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:53.423633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:53.423633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 624/1242] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:53.428456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4480f998-ec30-49dd-b5a8-ffa264988e5c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:53.521365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:53.527292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4480f998-ec30-49dd-b5a8-ffa264988e5c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:53.560270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:53.560871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:53.561153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:53.561427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:53.562092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:53.562398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:53.563756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:53.574458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 619/1243] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:53.580676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:53.658584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:53.678325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9f4e77-3b86-4a01-9efb-a5ea59f1d162 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:53.710785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:53.711399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:53.711741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:53.712013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:53.719024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:53.720493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:56:53.731081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:53.732184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:53.733393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 625/1244] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:56:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:53.738014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:53.834881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7bdea2-7131-4b3c-9c4a-52537eeb5712 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:53.867224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3ec504-fb09-4aba-85b6-fff7b6b8d6a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:53.867840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ec504-fb09-4aba-85b6-fff7b6b8d6a6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:53.868124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ec504-fb09-4aba-85b6-fff7b6b8d6a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:53.868393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3ec504-fb09-4aba-85b6-fff7b6b8d6a6 None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:53.869078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3ec504-fb09-4aba-85b6-fff7b6b8d6a6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:53.869385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3ec504-fb09-4aba-85b6-fff7b6b8d6a6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:53.870845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3ec504-fb09-4aba-85b6-fff7b6b8d6a6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:53.883794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 620/1245] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:56:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:53.890491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:53.987099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6560555a-f34c-4fca-9cab-2331bff7ab1e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:54.020642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d814f9-6174-484e-b28f-302656aa8ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:54.021234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d814f9-6174-484e-b28f-302656aa8ebe None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:54.021508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d814f9-6174-484e-b28f-302656aa8ebe None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:54.021804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d814f9-6174-484e-b28f-302656aa8ebe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:54.022514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d814f9-6174-484e-b28f-302656aa8ebe None admin] RBAC: Adding query filter params (role.id=c9f7a48328df4a7ea7a5981959f395f5, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=e1ffa92d7fb546399bc404053c201426) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:54.022815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d814f9-6174-484e-b28f-302656aa8ebe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:54.024211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d814f9-6174-484e-b28f-302656aa8ebe None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:54.029934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 626/1246] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:56:53 2022] GET /identity/v3/role_assignments?user.id=e1ffa92d7fb546399bc404053c201426&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=c9f7a48328df4a7ea7a5981959f395f5 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:55.976709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535179b3-1c2f-4720-b3e4-a8ed123fdc6d None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:55.977285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535179b3-1c2f-4720-b3e4-a8ed123fdc6d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:55.977285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535179b3-1c2f-4720-b3e4-a8ed123fdc6d None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:55.978620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 621/1247] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:56:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:56:55.983727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e55945-15df-4b4d-84c1-0321c31df1df None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:55.984120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e55945-15df-4b4d-84c1-0321c31df1df None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:55.984279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e55945-15df-4b4d-84c1-0321c31df1df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:55.990652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:56.002964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-60e55945-15df-4b4d-84c1-0321c31df1df None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:56:56.004135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-60e55945-15df-4b4d-84c1-0321c31df1df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:56:56.068022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 627/1248] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:56:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:56:56.336281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535179b3-1c2f-4720-b3e4-a8ed123fdc6d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:56.354546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:56.436228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535179b3-1c2f-4720-b3e4-a8ed123fdc6d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:56.470430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b4598e-3b12-44a9-8aae-16ca7aa6aba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:56.471057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b4598e-3b12-44a9-8aae-16ca7aa6aba2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:56.471336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b4598e-3b12-44a9-8aae-16ca7aa6aba2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:56.471607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b4598e-3b12-44a9-8aae-16ca7aa6aba2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:56.481458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b4598e-3b12-44a9-8aae-16ca7aa6aba2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-60b4598e-3b12-44a9-8aae-16ca7aa6aba2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:56:56.486243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:56.487414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:56.488549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 622/1249] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:56:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:56.493240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e55945-15df-4b4d-84c1-0321c31df1df None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:56.589761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e55945-15df-4b4d-84c1-0321c31df1df None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:56.618618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:56.623723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:56.624325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:56.624600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:56.624868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:56.625575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:56.625921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:56.627300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:56.635151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 628/1250] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:56:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:56.641614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b4598e-3b12-44a9-8aae-16ca7aa6aba2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:56.715923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:56.738391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b4598e-3b12-44a9-8aae-16ca7aa6aba2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:56.771580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:56.772248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:56.772526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:56.772797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:56.782470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:56.783901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:56.798053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:56.799180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:56.800350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:56.800350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:56.800350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:56.800350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 623/1251] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:56.805118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:56.901735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65ad0a-2031-46bb-b519-20da3e5fa970 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:56.933726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34360098-fb11-403a-8566-841542d8d92d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:56.934410 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34360098-fb11-403a-8566-841542d8d92d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:56.934671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34360098-fb11-403a-8566-841542d8d92d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:56.935462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34360098-fb11-403a-8566-841542d8d92d None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:56.936098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34360098-fb11-403a-8566-841542d8d92d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:56.936399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34360098-fb11-403a-8566-841542d8d92d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:56.937779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34360098-fb11-403a-8566-841542d8d92d None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:56.949393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 629/1252] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:56.955259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:57.054328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba5dff3-df5d-4052-bab7-dcd48aded7b2 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:57.079765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:57.087461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:57.088090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:57.088395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:57.088673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:57.096739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:57.098236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:56:57.110224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:57.111345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:56:57.112501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 624/1253] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:56 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:57.117243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34360098-fb11-403a-8566-841542d8d92d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:57.173155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:57.215391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34360098-fb11-403a-8566-841542d8d92d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:57.247932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:57.248527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:57.248801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:57.249069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:57.249885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:57.250190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:57.251587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:57.269492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 630/1254] 10.222.0.16 () {56 vars in 1143 bytes} [Thu Aug 25 04:56:57 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:57.275239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:57.375100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbba8a5a-733e-48d2-9ec7-116f317a236c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:57.408589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69cbf59-1dd2-4740-871c-389b3bbe9ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:57.409186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69cbf59-1dd2-4740-871c-389b3bbe9ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:57.409465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69cbf59-1dd2-4740-871c-389b3bbe9ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:57.409767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69cbf59-1dd2-4740-871c-389b3bbe9ec7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:57.463917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:57.475659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69cbf59-1dd2-4740-871c-389b3bbe9ec7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:57.477158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69cbf59-1dd2-4740-871c-389b3bbe9ec7 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:57.571087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 625/1255] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:56:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:56:57.576978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:57.674889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c71a8a-d395-4cef-bb6f-fd1a8d0e0b82 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:57.688755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:57.709728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:57.710628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:57.710909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:57.711183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:57.721025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:57.722433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:56:57.736987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:57.743221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:56:57.744354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:56:57.744354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:56:57.744354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:57.744354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 631/1256] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:56:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:57.744641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69cbf59-1dd2-4740-871c-389b3bbe9ec7 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:57.830398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:57.842367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69cbf59-1dd2-4740-871c-389b3bbe9ec7 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:57.875539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:57.876127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:57.876403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:57.876669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:57.877320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:57.877635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:57.878981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:57.890205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 626/1257] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:56:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:57.895782 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:57.994436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbdc1b7-9540-4e2f-9b4e-b1a47b4c033f None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:58.026909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:58.027531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:58.027809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:58.028080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:58.035109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:58.036530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:56:58.047480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:56:58.048519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:56:58.050153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 632/1258] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:56:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:56:58.054583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:58.153080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6847aae6-0d9c-423b-9ead-50a423339a60 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:58.185503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbff22e-a036-40a2-bce8-f0262738bf5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:58.186128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbff22e-a036-40a2-bce8-f0262738bf5b None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:58.186409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbff22e-a036-40a2-bce8-f0262738bf5b None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:58.186681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbff22e-a036-40a2-bce8-f0262738bf5b None admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:58.187363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbff22e-a036-40a2-bce8-f0262738bf5b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:58.187666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbff22e-a036-40a2-bce8-f0262738bf5b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:58.189079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbff22e-a036-40a2-bce8-f0262738bf5b None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:58.201183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 627/1259] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:56:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:56:58.207323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:56:58.234173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:56:58.306561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f288ca-33c0-4267-a8cf-a1b419a6e1e9 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:56:58.341427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b662318b-f3ef-45c4-b886-386db57a3598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:56:58.342055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b662318b-f3ef-45c4-b886-386db57a3598 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:56:58.342334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b662318b-f3ef-45c4-b886-386db57a3598 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:56:58.342605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b662318b-f3ef-45c4-b886-386db57a3598 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:56:58.343320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b662318b-f3ef-45c4-b886-386db57a3598 None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=e1ffa92d7fb546399bc404053c201426) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:56:58.343627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b662318b-f3ef-45c4-b886-386db57a3598 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:56:58.345053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b662318b-f3ef-45c4-b886-386db57a3598 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:56:58.350318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 633/1260] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:56:58 2022] GET /identity/v3/role_assignments?user.id=e1ffa92d7fb546399bc404053c201426&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:00.209797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1ab5d-eee5-48dc-8b44-5261e698ad3c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:00.210626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1ab5d-eee5-48dc-8b44-5261e698ad3c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:00.210626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1ab5d-eee5-48dc-8b44-5261e698ad3c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:00.211984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 628/1261] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:57:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:57:00.217148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dbe266-e10b-4be1-8263-4a09826917d8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:00.217630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dbe266-e10b-4be1-8263-4a09826917d8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:00.217739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dbe266-e10b-4be1-8263-4a09826917d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:00.237240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f5dbe266-e10b-4be1-8263-4a09826917d8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:57:00.238439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f5dbe266-e10b-4be1-8263-4a09826917d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:57:00.302917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 634/1262] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:57:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:57:00.573112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e1ab5d-eee5-48dc-8b44-5261e698ad3c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:00.584731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:00.672332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e1ab5d-eee5-48dc-8b44-5261e698ad3c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:00.705057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224fd87c-94b1-4503-82e0-3fdb85fdb70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:00.705705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224fd87c-94b1-4503-82e0-3fdb85fdb70c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:00.705986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224fd87c-94b1-4503-82e0-3fdb85fdb70c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:00.706256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224fd87c-94b1-4503-82e0-3fdb85fdb70c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:00.716653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224fd87c-94b1-4503-82e0-3fdb85fdb70c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-224fd87c-94b1-4503-82e0-3fdb85fdb70c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:57:00.721292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:00.722447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:57:00.723411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 629/1263] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:57:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:00.728364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dbe266-e10b-4be1-8263-4a09826917d8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:00.746632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:00.823799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dbe266-e10b-4be1-8263-4a09826917d8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:00.859691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:00.860396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:00.860680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:00.860951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:00.861637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:00.861946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:00.863312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:00.871578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 635/1264] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:57:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:00.877335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224fd87c-94b1-4503-82e0-3fdb85fdb70c None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:00.942665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:00.976047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224fd87c-94b1-4503-82e0-3fdb85fdb70c None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:01.007268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:01.007888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:01.008176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:01.008438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:01.020836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:01.022324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:57:01.036523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:01.037761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:57:01.038872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:57:01.038872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:57:01.038872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:57:01.038872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 630/1265] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:57:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:01.043627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:01.123301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:01.142279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75495e3-a912-4a54-b5fa-469e256ed7b7 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:01.175406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:01.176007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:01.176287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:01.176560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:01.177195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:01.177497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:01.178875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:01.189801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 636/1266] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:57:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:01.195618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:01.295406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a6c186-13bc-4e3f-bacb-c1d1cd72c726 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:01.314878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:01.329029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:01.329700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:01.329978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:01.330248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] PATH_INFO: `/v3/users/placement` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:01.337579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:01.339025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:57:01.349641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:01.350715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:57:01.351746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 631/1267] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:57:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:01.356506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:01.453354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab253a8-553a-455c-9ccd-5a42a2135b87 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:01.486971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:01.487573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:01.487849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:01.488119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:01.488841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:01.489173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:01.490652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:01.499892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:01.507598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 637/1268] 10.222.0.16 () {56 vars in 1143 bytes} [Thu Aug 25 04:57:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:01.513217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:01.609730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4498f7-d5a8-4aca-9777-8032c741eada None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:01.641764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:01.642380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:01.642653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:01.642917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:01.652852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:01.654332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:57:01.667645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:01.668761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:57:01.670378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:57:01.670378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:57:01.670378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:57:01.670378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 632/1269] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:57:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:01.675086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:01.773086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a7ba12-f595-41ce-ac52-0cf883c69222 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:01.806461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:01.807054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:01.807331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:01.807596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:01.808280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:01.808580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:01.810550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:01.822094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 638/1270] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:57:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:01.829887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:01.859866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:01.925304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bfdd30-a382-4d3a-bd99-82726192b156 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:01.958598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:01.959216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:01.959492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:01.959760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:01.966917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:01.968473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:57:01.979277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:01.980329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:57:01.981320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 633/1271] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:57:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:01.986253 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:02.043825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:02.083320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd09b1e6-16da-4f77-bbad-c70b7fc90b0c None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:02.114981 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4d4f98-50e5-4452-89b5-be69dcad8800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:02.115585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d4f98-50e5-4452-89b5-be69dcad8800 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:02.115863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d4f98-50e5-4452-89b5-be69dcad8800 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:02.116133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d4f98-50e5-4452-89b5-be69dcad8800 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:02.116814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4d4f98-50e5-4452-89b5-be69dcad8800 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:02.117119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4d4f98-50e5-4452-89b5-be69dcad8800 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:02.118561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4d4f98-50e5-4452-89b5-be69dcad8800 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:02.130699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 639/1272] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:57:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:02.136835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:02.228289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:02.236871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacaa3bc-6432-4fc7-a415-6a563ce77ddc None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:02.270644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78e1fbd-7d71-44ea-93d4-c7a4181a0c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:02.271353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78e1fbd-7d71-44ea-93d4-c7a4181a0c8c None admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:02.271636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78e1fbd-7d71-44ea-93d4-c7a4181a0c8c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:02.271912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78e1fbd-7d71-44ea-93d4-c7a4181a0c8c None admin] PATH_INFO: `/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/e1ffa92d7fb546399bc404053c201426/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:02.273870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78e1fbd-7d71-44ea-93d4-c7a4181a0c8c None admin] RBAC: Authorizing `identity:create_grant(project_id=14d7f359adfb449ab92725f20bfa3774, user_id=e1ffa92d7fb546399bc404053c201426, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:02.275325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78e1fbd-7d71-44ea-93d4-c7a4181a0c8c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:02.387399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 634/1273] 10.222.0.16 () {58 vars in 1439 bytes} [Thu Aug 25 04:57:02 2022] PUT /identity/v3/projects/14d7f359adfb449ab92725f20bfa3774/users/e1ffa92d7fb546399bc404053c201426/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 04:57:04.252336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40943b2-20e8-4046-8b7c-8747cdb7f7d0 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:04.252904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40943b2-20e8-4046-8b7c-8747cdb7f7d0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:04.252904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40943b2-20e8-4046-8b7c-8747cdb7f7d0 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:04.254291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 640/1274] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:57:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:57:04.259497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b60cc3-4cbe-4105-a600-5c6de5fdc03b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:04.259895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b60cc3-4cbe-4105-a600-5c6de5fdc03b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:04.260129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b60cc3-4cbe-4105-a600-5c6de5fdc03b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:04.278251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-59b60cc3-4cbe-4105-a600-5c6de5fdc03b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:57:04.279409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-59b60cc3-4cbe-4105-a600-5c6de5fdc03b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:57:04.344907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 635/1275] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:57:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:57:04.613061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40943b2-20e8-4046-8b7c-8747cdb7f7d0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:04.713231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40943b2-20e8-4046-8b7c-8747cdb7f7d0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:04.730546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:04.746525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f895df8-4074-436c-9997-821e6390c59d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:04.747206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f895df8-4074-436c-9997-821e6390c59d None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:04.747485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f895df8-4074-436c-9997-821e6390c59d None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:04.747755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f895df8-4074-436c-9997-821e6390c59d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:04.757468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f895df8-4074-436c-9997-821e6390c59d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-7f895df8-4074-436c-9997-821e6390c59d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 25 04:57:04.762121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:04.767613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:04.768740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 641/1276] 10.222.0.16 () {56 vars in 1096 bytes} [Thu Aug 25 04:57:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:04.769666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b60cc3-4cbe-4105-a600-5c6de5fdc03b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:04.848663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:04.866000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b60cc3-4cbe-4105-a600-5c6de5fdc03b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:04.898356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:04.898957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:04.899233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:04.899502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:04.900151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:04.900452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:04.901839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:04.910221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 636/1277] 10.222.0.16 () {56 vars in 1099 bytes} [Thu Aug 25 04:57:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:04.916278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f895df8-4074-436c-9997-821e6390c59d None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:05.017382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f895df8-4074-436c-9997-821e6390c59d None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:05.050733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:05.051359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:05.051638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:05.051913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:05.062237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:05.063702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:57:05.078015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:05.090224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:57:05.091850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:57:05.091850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:57:05.091850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:05.091850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 642/1278] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:57:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:05.092132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:05.182901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ec6801-04d0-421b-ab99-996219710250 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:05.216886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:05.217507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:05.217815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:05.218108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:05.218733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:05.219037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:05.220418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:05.232727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 637/1279] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:57:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:05.238416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:05.269831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:05.336284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e69e347-e079-4975-9d07-9e384c93add4 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:05.368679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:05.369579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:05.370087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:05.370940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:05.378562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:05.379974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 25 04:57:05.391530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:05.392645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:05.393731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 643/1280] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:57:05 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:05.398476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:05.402249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:05.492702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789ffd0d-42c7-49f6-91f9-cbe9867e9766 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:05.525167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:05.525798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:05.526079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:05.526352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:05.527066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:05.527373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:05.528944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:05.546098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 638/1281] 10.222.0.16 () {56 vars in 1143 bytes} [Thu Aug 25 04:57:05 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:05.551858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:05.642285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:05.651595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532a1f9b-b540-45c9-bc95-a53afb43b352 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:05.686291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cd1040-72ca-448c-95b1-223b50aafe51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:05.686894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cd1040-72ca-448c-95b1-223b50aafe51 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:05.695027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cd1040-72ca-448c-95b1-223b50aafe51 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:05.695027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cd1040-72ca-448c-95b1-223b50aafe51 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:05.752757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cd1040-72ca-448c-95b1-223b50aafe51 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:05.754174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cd1040-72ca-448c-95b1-223b50aafe51 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:05.838810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:05.848974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 644/1282] 10.222.0.16 () {58 vars in 1282 bytes} [Thu Aug 25 04:57:05 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 25 04:57:05.854923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:05.915306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:05.952918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a264da2-2cca-4d06-a35a-8d97f731945d None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:05.988256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:05.988883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:05.989163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:05.989436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:05.999260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:06.000690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 25 04:57:06.015240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:06.016282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 25 04:57:06.017303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 25 04:57:06.017303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 25 04:57:06.017303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:57:06.017303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 639/1283] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 04:57:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:06.022339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cd1040-72ca-448c-95b1-223b50aafe51 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:06.119853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cd1040-72ca-448c-95b1-223b50aafe51 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:06.152725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:06.156379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:06.156379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:06.156379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:06.156379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:06.156379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:06.156811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:06.167738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 645/1284] 10.222.0.16 () {56 vars in 1109 bytes} [Thu Aug 25 04:57:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:06.173529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:06.272147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee74cab8-be40-4b5d-a738-ec2bc2d6de9e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:06.285926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:06.305129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:06.305915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:06.306196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:06.306470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:06.312939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:06.314409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 25 04:57:06.325959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:06.326866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: ERROR keystone.server.flask.application Aug 25 04:57:06.327834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 640/1285] 10.222.0.16 () {56 vars in 1111 bytes} [Thu Aug 25 04:57:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:06.332999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:06.382190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:06.429640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81fa8ee-5baa-4abc-8f5d-f3dfb3b80383 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:06.462014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3421c-33f6-4f97-967d-2fba6eb37031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:06.462612 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3421c-33f6-4f97-967d-2fba6eb37031 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:06.462889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3421c-33f6-4f97-967d-2fba6eb37031 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:06.463157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3421c-33f6-4f97-967d-2fba6eb37031 None admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:06.463835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3421c-33f6-4f97-967d-2fba6eb37031 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:06.464138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3421c-33f6-4f97-967d-2fba6eb37031 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:06.466331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3421c-33f6-4f97-967d-2fba6eb37031 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:06.478621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 646/1286] 10.222.0.16 () {56 vars in 1148 bytes} [Thu Aug 25 04:57:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:06.484777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:06.583342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be2bf5-4fc9-4353-a948-f8a01571ffc5 None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:06.616057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b675a0c7-157d-42fa-ad74-ad22c7592b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:06.616656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b675a0c7-157d-42fa-ad74-ad22c7592b4c None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:06.616933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b675a0c7-157d-42fa-ad74-ad22c7592b4c None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:06.617196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b675a0c7-157d-42fa-ad74-ad22c7592b4c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:06.617981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b675a0c7-157d-42fa-ad74-ad22c7592b4c None admin] RBAC: Adding query filter params (role.id=301b10bcbece4095821ce8c90c3e3332, scope.project.id=14d7f359adfb449ab92725f20bfa3774, user.id=e1ffa92d7fb546399bc404053c201426) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:06.618286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b675a0c7-157d-42fa-ad74-ad22c7592b4c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:06.619747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b675a0c7-157d-42fa-ad74-ad22c7592b4c None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:06.625970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 641/1287] 10.222.0.16 () {56 vars in 1374 bytes} [Thu Aug 25 04:57:06 2022] GET /identity/v3/role_assignments?user.id=e1ffa92d7fb546399bc404053c201426&scope.project.id=14d7f359adfb449ab92725f20bfa3774&role.id=301b10bcbece4095821ce8c90c3e3332 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:08.536725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e35fcc-e988-46b5-ba8d-77f4ac9756ab None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:08.537226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e35fcc-e988-46b5-ba8d-77f4ac9756ab None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:08.537226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e35fcc-e988-46b5-ba8d-77f4ac9756ab None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:08.538561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 647/1288] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:57:08.543749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8037e22e-ea4d-49ca-a836-f15683ebf331 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:08.544144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8037e22e-ea4d-49ca-a836-f15683ebf331 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:08.544294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8037e22e-ea4d-49ca-a836-f15683ebf331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:08.562194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8037e22e-ea4d-49ca-a836-f15683ebf331 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:57:08.563410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8037e22e-ea4d-49ca-a836-f15683ebf331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:57:08.593890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:08.630321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 642/1289] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:57:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:57:08.897582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e35fcc-e988-46b5-ba8d-77f4ac9756ab None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:08.997365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e35fcc-e988-46b5-ba8d-77f4ac9756ab None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:09.007971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:09.031578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:09.032194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:09.032467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:09.032734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:09.033300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:09.034681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:09.043302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:09.044490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:09.045360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 648/1290] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:57:08 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:09.050476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8037e22e-ea4d-49ca-a836-f15683ebf331 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:09.154447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8037e22e-ea4d-49ca-a836-f15683ebf331 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:09.188383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8582d1cd-f677-4015-9d9d-74d7ede6fb69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:09.188979 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582d1cd-f677-4015-9d9d-74d7ede6fb69 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:09.189255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582d1cd-f677-4015-9d9d-74d7ede6fb69 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:09.189523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8582d1cd-f677-4015-9d9d-74d7ede6fb69 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:09.190167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8582d1cd-f677-4015-9d9d-74d7ede6fb69 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:09.190463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8582d1cd-f677-4015-9d9d-74d7ede6fb69 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:09.191812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8582d1cd-f677-4015-9d9d-74d7ede6fb69 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:09.196478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 643/1291] 10.222.0.16 () {56 vars in 1116 bytes} [Thu Aug 25 04:57:09 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:09.202189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:09.304698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9d78a8-ee13-4213-9188-bb1ff1418c74 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:09.338069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecd8a8d-7847-4d80-835e-3c47136ff5d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:09.338660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd8a8d-7847-4d80-835e-3c47136ff5d5 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:09.338934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd8a8d-7847-4d80-835e-3c47136ff5d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:09.339200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd8a8d-7847-4d80-835e-3c47136ff5d5 None admin] PATH_INFO: `/v3/services` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:09.339867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecd8a8d-7847-4d80-835e-3c47136ff5d5 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:09.340213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecd8a8d-7847-4d80-835e-3c47136ff5d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:09.341614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aecd8a8d-7847-4d80-835e-3c47136ff5d5 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:09.345921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 649/1292] 10.222.0.16 () {56 vars in 1116 bytes} [Thu Aug 25 04:57:09 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:11.231565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6150982-89c6-4d99-88df-5a3a7ae5849d None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:11.231993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6150982-89c6-4d99-88df-5a3a7ae5849d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:11.232121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6150982-89c6-4d99-88df-5a3a7ae5849d None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:11.233459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 644/1293] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:57:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:57:11.238598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca1df6b-f113-498c-b91c-0b262c222178 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:11.239048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca1df6b-f113-498c-b91c-0b262c222178 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:11.239143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca1df6b-f113-498c-b91c-0b262c222178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:11.257448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1ca1df6b-f113-498c-b91c-0b262c222178 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:57:11.258719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1ca1df6b-f113-498c-b91c-0b262c222178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:57:11.288188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:11.323609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 650/1294] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:57:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:57:11.592088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6150982-89c6-4d99-88df-5a3a7ae5849d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:11.622474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:11.690896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6150982-89c6-4d99-88df-5a3a7ae5849d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:11.724554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f156e435-9eb8-4001-96fe-66f744c1b524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:11.725146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f156e435-9eb8-4001-96fe-66f744c1b524 None admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:11.725420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f156e435-9eb8-4001-96fe-66f744c1b524 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:11.725724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f156e435-9eb8-4001-96fe-66f744c1b524 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:11.726356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f156e435-9eb8-4001-96fe-66f744c1b524 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:11.727709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f156e435-9eb8-4001-96fe-66f744c1b524 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:11.802832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 645/1295] 10.222.0.16 () {60 vars in 1141 bytes} [Thu Aug 25 04:57:11 2022] POST /identity/v3/services => generated 246 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:57:13.706544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f22948-a809-4252-83b7-39dc116b1f59 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:13.707132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f22948-a809-4252-83b7-39dc116b1f59 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:13.707132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f22948-a809-4252-83b7-39dc116b1f59 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:13.708432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 651/1296] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:57:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:57:13.713941 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933d7e5-b4b7-4cc1-99bd-d812b811256c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:13.714388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933d7e5-b4b7-4cc1-99bd-d812b811256c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:13.714488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f933d7e5-b4b7-4cc1-99bd-d812b811256c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:13.732550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f933d7e5-b4b7-4cc1-99bd-d812b811256c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:57:13.733755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f933d7e5-b4b7-4cc1-99bd-d812b811256c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:57:13.804168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 646/1297] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:57:13 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:57:14.069114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f22948-a809-4252-83b7-39dc116b1f59 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:14.172022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f22948-a809-4252-83b7-39dc116b1f59 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:14.204785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:14.205411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:14.205723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:14.206000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:14.206578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:14.207959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:14.217310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:14.218569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:14.219509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 652/1298] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:57:14 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:14.225685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933d7e5-b4b7-4cc1-99bd-d812b811256c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:14.243484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:14.321342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f933d7e5-b4b7-4cc1-99bd-d812b811256c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:14.356328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:14.357004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:14.357288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:14.357562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:14.358227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] RBAC: Adding query filter params (name=placement) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:14.358534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:14.359901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:14.365125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 647/1299] 10.222.0.16 () {56 vars in 1116 bytes} [Thu Aug 25 04:57:14 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:14.371090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:14.406623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:14.475123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1370867e-7587-4f18-b0b5-b04762889a16 None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:14.507112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4d11f1-64c9-4fd4-96e8-4ec1644b0752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:14.507706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4d11f1-64c9-4fd4-96e8-4ec1644b0752 None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:14.507989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4d11f1-64c9-4fd4-96e8-4ec1644b0752 None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:14.508259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4d11f1-64c9-4fd4-96e8-4ec1644b0752 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:14.508947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4d11f1-64c9-4fd4-96e8-4ec1644b0752 None admin] RBAC: Adding query filter params (interface=public, service_id=df90d91fc3ef478db9c2146ddcec2182, region_id=RegionOne) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:14.509250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4d11f1-64c9-4fd4-96e8-4ec1644b0752 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:14.510631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4d11f1-64c9-4fd4-96e8-4ec1644b0752 None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:14.515949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 653/1300] 10.222.0.16 () {56 vars in 1251 bytes} [Thu Aug 25 04:57:14 2022] GET /identity/v3/endpoints?service_id=df90d91fc3ef478db9c2146ddcec2182&interface=public®ion_id=RegionOne => generated 194 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:14.521671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:14.600968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:14.618173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6466d5b-21c8-4a06-8d52-31014121212e None admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:14.650499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe382d9d-9652-4d27-932a-490fd5703315 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:14.651095 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe382d9d-9652-4d27-932a-490fd5703315 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:14.651375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe382d9d-9652-4d27-932a-490fd5703315 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:14.651653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe382d9d-9652-4d27-932a-490fd5703315 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:14.652235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe382d9d-9652-4d27-932a-490fd5703315 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:14.654186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe382d9d-9652-4d27-932a-490fd5703315 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:14.660684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 648/1301] 10.222.0.16 () {56 vars in 1087 bytes} [Thu Aug 25 04:57:14 2022] GET /identity/v3/services => generated 2191 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 04:57:16.554190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1a90ca-99fa-4fc7-bf57-409ef9e0781a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:16.554720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1a90ca-99fa-4fc7-bf57-409ef9e0781a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:16.554720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1a90ca-99fa-4fc7-bf57-409ef9e0781a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:16.556008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 654/1302] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 04:57:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 04:57:16.561744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81f100-6550-43ea-8c03-685ae284085d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:16.562193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81f100-6550-43ea-8c03-685ae284085d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:16.562288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b81f100-6550-43ea-8c03-685ae284085d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:16.580784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3b81f100-6550-43ea-8c03-685ae284085d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 04:57:16.585065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3b81f100-6550-43ea-8c03-685ae284085d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 04:57:16.646872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 649/1303] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 04:57:16 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 25 04:57:16.916104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1a90ca-99fa-4fc7-bf57-409ef9e0781a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:17.008668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:17.015608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1a90ca-99fa-4fc7-bf57-409ef9e0781a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:17.047162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:17.047776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:17.048054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:17.048325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] PATH_INFO: `/v3/services/placement` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:17.048895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:17.050275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 25 04:57:17.059203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 25 04:57:17.060193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.get_or_create( Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application with Lock( Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self._enter() Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application return self.creator() Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application created_value = creator( Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: ERROR keystone.server.flask.application Aug 25 04:57:17.061162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 655/1304] 10.222.0.16 () {56 vars in 1117 bytes} [Thu Aug 25 04:57:16 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 25 04:57:17.066259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81f100-6550-43ea-8c03-685ae284085d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:17.141527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:17.163154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b81f100-6550-43ea-8c03-685ae284085d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:17.195369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735d60d1-dc43-42bb-9ad4-7a74a8995cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:17.195990 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d60d1-dc43-42bb-9ad4-7a74a8995cf6 None admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:17.196229 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d60d1-dc43-42bb-9ad4-7a74a8995cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:17.196499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735d60d1-dc43-42bb-9ad4-7a74a8995cf6 None admin] PATH_INFO: `/v3/services` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:17.197118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735d60d1-dc43-42bb-9ad4-7a74a8995cf6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 04:57:17.197420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735d60d1-dc43-42bb-9ad4-7a74a8995cf6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:17.198801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735d60d1-dc43-42bb-9ad4-7a74a8995cf6 None admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:17.204287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 650/1305] 10.222.0.16 () {56 vars in 1116 bytes} [Thu Aug 25 04:57:17 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 04:57:17.210464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 04:57:17.308038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a915f9-0f34-4089-809e-e1a9ee5cfe6e None admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 04:57:17.342058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ac83d-a41c-4750-b5cd-ad4d98cdbdcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 04:57:17.342695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ac83d-a41c-4750-b5cd-ad4d98cdbdcb None admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 04:57:17.342973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ac83d-a41c-4750-b5cd-ad4d98cdbdcb None admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 04:57:17.343273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ac83d-a41c-4750-b5cd-ad4d98cdbdcb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 04:57:17.343936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474ac83d-a41c-4750-b5cd-ad4d98cdbdcb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 04:57:17.345294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474ac83d-a41c-4750-b5cd-ad4d98cdbdcb None admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 04:57:17.460918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 656/1306] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 04:57:17 2022] POST /identity/v3/endpoints => generated 331 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 04:57:27.283948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=30 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 04:57:27.455451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:04:59.396908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a4eba-87ca-4dc3-89b7-ffe4c674e88a None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:04:59.397547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a4eba-87ca-4dc3-89b7-ffe4c674e88a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:04:59.397547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a4eba-87ca-4dc3-89b7-ffe4c674e88a None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:04:59.404380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920f510e-b509-4d72-9434-555c165f26ba None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:04:59.404845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920f510e-b509-4d72-9434-555c165f26ba None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:04:59.404929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920f510e-b509-4d72-9434-555c165f26ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:04:59.409524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 651/1307] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:04:59 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:04:59.498339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-920f510e-b509-4d72-9434-555c165f26ba None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:04:59.499460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-920f510e-b509-4d72-9434-555c165f26ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:04:59.716700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 657/1308] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:04:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:05:00.011137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c253e70-d323-46f7-95ca-ce0c89f66962 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:00.011430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c253e70-d323-46f7-95ca-ce0c89f66962 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:00.011620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c253e70-d323-46f7-95ca-ce0c89f66962 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:00.012951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 652/1309] 10.222.0.16 () {54 vars in 965 bytes} [Thu Aug 25 05:05:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:00.019409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31586b0a-ab55-4d38-81f2-dbf17cf10fe2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:00.019699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31586b0a-ab55-4d38-81f2-dbf17cf10fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:00.019975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31586b0a-ab55-4d38-81f2-dbf17cf10fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:00.098974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-31586b0a-ab55-4d38-81f2-dbf17cf10fe2 None None] MFA Rules not processed for user `9220ba3637a542a382ad1d42ff1ff459`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:00.100177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-31586b0a-ab55-4d38-81f2-dbf17cf10fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:00.122063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:00.185801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 658/1310] 10.222.0.16 () {58 vars in 1062 bytes} [Thu Aug 25 05:05:00 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:00.191671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c253e70-d323-46f7-95ca-ce0c89f66962 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:00.193635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4c253e70-d323-46f7-95ca-ce0c89f66962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:00.318350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c253e70-d323-46f7-95ca-ce0c89f66962 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:00.399338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7319b181-1b87-4d0c-b7e0-aca706aa490d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:00.400043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7319b181-1b87-4d0c-b7e0-aca706aa490d service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:00.400365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7319b181-1b87-4d0c-b7e0-aca706aa490d service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:00.400551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7319b181-1b87-4d0c-b7e0-aca706aa490d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:00.402170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7319b181-1b87-4d0c-b7e0-aca706aa490d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:00.463222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:00.522666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7319b181-1b87-4d0c-b7e0-aca706aa490d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:00.524101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7319b181-1b87-4d0c-b7e0-aca706aa490d service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:00.543208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 653/1311] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:02.733907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37fe875-58d8-4ff7-8663-6b452dabe355 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:02.734447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37fe875-58d8-4ff7-8663-6b452dabe355 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:02.734669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37fe875-58d8-4ff7-8663-6b452dabe355 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:02.736329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 659/1312] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:02.741812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb4e93c-65f2-4059-a0e9-7fa019959dd9 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:02.742693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb4e93c-65f2-4059-a0e9-7fa019959dd9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:02.742997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb4e93c-65f2-4059-a0e9-7fa019959dd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:02.823485 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-fcb4e93c-65f2-4059-a0e9-7fa019959dd9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:02.824721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-fcb4e93c-65f2-4059-a0e9-7fa019959dd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:02.930959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 654/1313] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:03.210305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37fe875-58d8-4ff7-8663-6b452dabe355 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:03.211904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d37fe875-58d8-4ff7-8663-6b452dabe355 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:03.315674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37fe875-58d8-4ff7-8663-6b452dabe355 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:03.386646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:03.389871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f094723a-b222-4a4b-bd8e-009565ca98ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:03.390482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f094723a-b222-4a4b-bd8e-009565ca98ca service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:03.390762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f094723a-b222-4a4b-bd8e-009565ca98ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:03.391034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f094723a-b222-4a4b-bd8e-009565ca98ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:03.392637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f094723a-b222-4a4b-bd8e-009565ca98ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:03.504527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f094723a-b222-4a4b-bd8e-009565ca98ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:03.505973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f094723a-b222-4a4b-bd8e-009565ca98ca service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:03.525340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 660/1314] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:05.924955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51a9a95-170d-420c-b5d4-3ce9d3364b6f None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:05.925491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51a9a95-170d-420c-b5d4-3ce9d3364b6f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:05.925491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51a9a95-170d-420c-b5d4-3ce9d3364b6f None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:05.926975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 655/1315] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:05.932009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b126f8-b4c4-4001-8b76-9ca6b33d3892 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:05.932416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b126f8-b4c4-4001-8b76-9ca6b33d3892 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:05.932555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b126f8-b4c4-4001-8b76-9ca6b33d3892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:05.996550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-37b126f8-b4c4-4001-8b76-9ca6b33d3892 None None] MFA Rules not processed for user `055c8932e5b14530ad56960260a63d7e`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:05.997763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-37b126f8-b4c4-4001-8b76-9ca6b33d3892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:06.049112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:06.091675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 661/1316] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:05 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:06.373255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51a9a95-170d-420c-b5d4-3ce9d3364b6f None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:06.374873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b51a9a95-170d-420c-b5d4-3ce9d3364b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:06.384620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:06.478669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51a9a95-170d-420c-b5d4-3ce9d3364b6f None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:06.550177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc705d-8488-4594-8c80-70288e87b66e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:06.550780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc705d-8488-4594-8c80-70288e87b66e service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:06.551057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc705d-8488-4594-8c80-70288e87b66e service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:06.551325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc705d-8488-4594-8c80-70288e87b66e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:06.552905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0fbc705d-8488-4594-8c80-70288e87b66e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:06.613612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:06.659673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbc705d-8488-4594-8c80-70288e87b66e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:06.661093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbc705d-8488-4594-8c80-70288e87b66e service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:06.678874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 656/1317] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:06 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:09.305342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840fe801-a465-42a2-8d7d-1aaeb44e44f1 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:09.305926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840fe801-a465-42a2-8d7d-1aaeb44e44f1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:09.305926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840fe801-a465-42a2-8d7d-1aaeb44e44f1 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:09.307323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 662/1318] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:09.313142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d150030-a807-4f2d-a68e-0c6aa4bfae0c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:09.313565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d150030-a807-4f2d-a68e-0c6aa4bfae0c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:09.313714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d150030-a807-4f2d-a68e-0c6aa4bfae0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:09.378826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-0d150030-a807-4f2d-a68e-0c6aa4bfae0c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:09.380023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0d150030-a807-4f2d-a68e-0c6aa4bfae0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:09.473134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 657/1319] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:09 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:09.753239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840fe801-a465-42a2-8d7d-1aaeb44e44f1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:09.754822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-840fe801-a465-42a2-8d7d-1aaeb44e44f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:09.973272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840fe801-a465-42a2-8d7d-1aaeb44e44f1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:10.044221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85486b62-613f-49f4-91d9-6eb056e1815f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:10.044757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85486b62-613f-49f4-91d9-6eb056e1815f service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:10.045033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85486b62-613f-49f4-91d9-6eb056e1815f service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:10.045300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85486b62-613f-49f4-91d9-6eb056e1815f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:10.046900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-85486b62-613f-49f4-91d9-6eb056e1815f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:10.053609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:10.151296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85486b62-613f-49f4-91d9-6eb056e1815f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:10.152726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85486b62-613f-49f4-91d9-6eb056e1815f service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:10.170581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 663/1320] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:12.924534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f4b205-118c-401c-b482-6c46d9b6a900 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:12.925070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f4b205-118c-401c-b482-6c46d9b6a900 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:12.925070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f4b205-118c-401c-b482-6c46d9b6a900 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:12.926451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 658/1321] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:12.931715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c809149-1f82-4c5b-8543-2ac325d7af91 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:12.932162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c809149-1f82-4c5b-8543-2ac325d7af91 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:12.932267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c809149-1f82-4c5b-8543-2ac325d7af91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:12.999753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5c809149-1f82-4c5b-8543-2ac325d7af91 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:13.000955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5c809149-1f82-4c5b-8543-2ac325d7af91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:13.058167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:13.098261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 664/1322] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:13.379199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f4b205-118c-401c-b482-6c46d9b6a900 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:13.380792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a6f4b205-118c-401c-b482-6c46d9b6a900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:13.402219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:13.489388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f4b205-118c-401c-b482-6c46d9b6a900 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:13.565104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4775da67-500a-4ffa-9e31-30b60807b6d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:13.565750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4775da67-500a-4ffa-9e31-30b60807b6d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:13.566034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4775da67-500a-4ffa-9e31-30b60807b6d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:13.566306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4775da67-500a-4ffa-9e31-30b60807b6d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:13.567973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4775da67-500a-4ffa-9e31-30b60807b6d7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:13.644808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:13.681976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4775da67-500a-4ffa-9e31-30b60807b6d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:13.683409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4775da67-500a-4ffa-9e31-30b60807b6d7 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:13.702119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 659/1323] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:15.654237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca114070-981e-4502-8907-ad5f489ca595 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:15.654759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca114070-981e-4502-8907-ad5f489ca595 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:15.654759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca114070-981e-4502-8907-ad5f489ca595 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:15.656267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 665/1324] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:15.661224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fc5de-b73a-4ed0-b300-9f2945e97186 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:15.661628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fc5de-b73a-4ed0-b300-9f2945e97186 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:15.661847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fc5de-b73a-4ed0-b300-9f2945e97186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:15.726986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-d77fc5de-b73a-4ed0-b300-9f2945e97186 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:15.728102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d77fc5de-b73a-4ed0-b300-9f2945e97186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:15.823118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 660/1325] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:16.104772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca114070-981e-4502-8907-ad5f489ca595 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:16.106990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ca114070-981e-4502-8907-ad5f489ca595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:16.213534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca114070-981e-4502-8907-ad5f489ca595 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:16.290736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f67048-e1db-4783-9b66-783bff37b295 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:16.298145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f67048-e1db-4783-9b66-783bff37b295 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:16.298145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f67048-e1db-4783-9b66-783bff37b295 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:16.298145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f67048-e1db-4783-9b66-783bff37b295 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:16.298145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-89f67048-e1db-4783-9b66-783bff37b295 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:16.316126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:16.409265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f67048-e1db-4783-9b66-783bff37b295 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:16.410727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f67048-e1db-4783-9b66-783bff37b295 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:16.428860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 666/1326] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:18.369341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4923653a-9d1e-434e-83bf-d6b079ec90d6 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:18.369929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4923653a-9d1e-434e-83bf-d6b079ec90d6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:18.369929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4923653a-9d1e-434e-83bf-d6b079ec90d6 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:18.371469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 661/1327] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:18.376539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236050a1-af1b-4b79-bff6-e3a2ee62e2f3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:18.376949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236050a1-af1b-4b79-bff6-e3a2ee62e2f3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:18.377074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236050a1-af1b-4b79-bff6-e3a2ee62e2f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:18.444992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-236050a1-af1b-4b79-bff6-e3a2ee62e2f3 None None] MFA Rules not processed for user `055c8932e5b14530ad56960260a63d7e`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:18.446220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-236050a1-af1b-4b79-bff6-e3a2ee62e2f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:18.513626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:18.547592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 667/1328] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:18 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:18.831185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4923653a-9d1e-434e-83bf-d6b079ec90d6 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:18.832746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4923653a-9d1e-434e-83bf-d6b079ec90d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:18.863436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:18.967005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4923653a-9d1e-434e-83bf-d6b079ec90d6 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:19.053935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6da13b7-7bb8-40ef-8f30-668721dd8ec9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:19.054540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da13b7-7bb8-40ef-8f30-668721dd8ec9 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:19.054828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da13b7-7bb8-40ef-8f30-668721dd8ec9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:19.055088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da13b7-7bb8-40ef-8f30-668721dd8ec9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:19.056673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b6da13b7-7bb8-40ef-8f30-668721dd8ec9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:19.137576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:19.171317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6da13b7-7bb8-40ef-8f30-668721dd8ec9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:19.172839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6da13b7-7bb8-40ef-8f30-668721dd8ec9 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:19.192038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 662/1329] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:18 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:21.690417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbbfd36-46af-4d24-85a0-89a96f773fa0 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:21.690965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbbfd36-46af-4d24-85a0-89a96f773fa0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:21.690965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbbfd36-46af-4d24-85a0-89a96f773fa0 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:21.692320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 668/1330] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:21.697476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8378841-4e47-4ab3-aa1e-e5564eee9aff None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:21.697940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8378841-4e47-4ab3-aa1e-e5564eee9aff None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:21.698051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8378841-4e47-4ab3-aa1e-e5564eee9aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:21.763710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e8378841-4e47-4ab3-aa1e-e5564eee9aff None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:21.764905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e8378841-4e47-4ab3-aa1e-e5564eee9aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:21.859717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 663/1331] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:22.139646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbbfd36-46af-4d24-85a0-89a96f773fa0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:22.141190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6fbbfd36-46af-4d24-85a0-89a96f773fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:22.243091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbbfd36-46af-4d24-85a0-89a96f773fa0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:22.314920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c414de6-7f6c-4d05-aa5a-a8e6a2b9fc1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:22.315520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c414de6-7f6c-4d05-aa5a-a8e6a2b9fc1b service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:22.315794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c414de6-7f6c-4d05-aa5a-a8e6a2b9fc1b service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:22.316061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c414de6-7f6c-4d05-aa5a-a8e6a2b9fc1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:22.317650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9c414de6-7f6c-4d05-aa5a-a8e6a2b9fc1b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:22.344816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:22.425289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c414de6-7f6c-4d05-aa5a-a8e6a2b9fc1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:22.426679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c414de6-7f6c-4d05-aa5a-a8e6a2b9fc1b service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:22.444669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 669/1332] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:24.379391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37bc19a-cbf3-4f99-ab91-0b545537e08a None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:24.379913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37bc19a-cbf3-4f99-ab91-0b545537e08a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:24.379913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37bc19a-cbf3-4f99-ab91-0b545537e08a None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:24.381280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 664/1333] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:24.386521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4547a37-f016-48f3-8513-e75e3b4239fb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:24.386901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4547a37-f016-48f3-8513-e75e3b4239fb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:24.387077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4547a37-f016-48f3-8513-e75e3b4239fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:24.451808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e4547a37-f016-48f3-8513-e75e3b4239fb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:24.453002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e4547a37-f016-48f3-8513-e75e3b4239fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:24.525075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:24.547723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 670/1334] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:24.845412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37bc19a-cbf3-4f99-ab91-0b545537e08a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:24.846989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d37bc19a-cbf3-4f99-ab91-0b545537e08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:24.887443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:24.951033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37bc19a-cbf3-4f99-ab91-0b545537e08a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:25.023824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3c182f-a397-4303-82e7-cabbdfcb9dfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:25.024448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c182f-a397-4303-82e7-cabbdfcb9dfd service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:25.024727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c182f-a397-4303-82e7-cabbdfcb9dfd service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:25.024999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3c182f-a397-4303-82e7-cabbdfcb9dfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:25.026615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7a3c182f-a397-4303-82e7-cabbdfcb9dfd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:25.110892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:25.135527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3c182f-a397-4303-82e7-cabbdfcb9dfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:25.137461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3c182f-a397-4303-82e7-cabbdfcb9dfd service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:25.155512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 665/1335] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:27.682229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fc961b-ff34-4711-bb1d-ff8088e6096d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:27.682795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fc961b-ff34-4711-bb1d-ff8088e6096d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:27.682795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fc961b-ff34-4711-bb1d-ff8088e6096d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:27.684650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 671/1336] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:27.689742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35301e2-c6d7-4538-a252-ea6017a7e49a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:27.690202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35301e2-c6d7-4538-a252-ea6017a7e49a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:27.690360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35301e2-c6d7-4538-a252-ea6017a7e49a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:27.755729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e35301e2-c6d7-4538-a252-ea6017a7e49a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:27.756923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e35301e2-c6d7-4538-a252-ea6017a7e49a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:27.851243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 666/1337] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:27 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:28.132163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fc961b-ff34-4711-bb1d-ff8088e6096d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:28.133716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d4fc961b-ff34-4711-bb1d-ff8088e6096d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:28.235784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fc961b-ff34-4711-bb1d-ff8088e6096d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:28.308979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74199d1-7ab8-45aa-b977-5760e1aa19b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:28.309631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74199d1-7ab8-45aa-b977-5760e1aa19b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:28.309909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74199d1-7ab8-45aa-b977-5760e1aa19b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:28.310183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74199d1-7ab8-45aa-b977-5760e1aa19b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:28.311763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b74199d1-7ab8-45aa-b977-5760e1aa19b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:28.346925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:28.419409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74199d1-7ab8-45aa-b977-5760e1aa19b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:28.420769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74199d1-7ab8-45aa-b977-5760e1aa19b6 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:28.438962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 672/1338] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:28 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:32.437166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea343035-f8c1-4cca-bb02-938d7efca6c5 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:32.437755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea343035-f8c1-4cca-bb02-938d7efca6c5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:32.437865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea343035-f8c1-4cca-bb02-938d7efca6c5 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:32.439283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 667/1339] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:32.444719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffe2c44-fe58-4185-a6b8-6c852555abb9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:32.445188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffe2c44-fe58-4185-a6b8-6c852555abb9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:32.445297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffe2c44-fe58-4185-a6b8-6c852555abb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:32.512128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fffe2c44-fe58-4185-a6b8-6c852555abb9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:32.513361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fffe2c44-fe58-4185-a6b8-6c852555abb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:32.599678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:32.611108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 673/1340] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:32.900046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea343035-f8c1-4cca-bb02-938d7efca6c5 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:32.901975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ea343035-f8c1-4cca-bb02-938d7efca6c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:32.950686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:33.006566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea343035-f8c1-4cca-bb02-938d7efca6c5 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:33.078920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd77871-a66d-4b77-ad81-1d47f6b3fe5c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:33.079628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd77871-a66d-4b77-ad81-1d47f6b3fe5c service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:33.079916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd77871-a66d-4b77-ad81-1d47f6b3fe5c service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:33.080191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd77871-a66d-4b77-ad81-1d47f6b3fe5c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:33.081833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-dfd77871-a66d-4b77-ad81-1d47f6b3fe5c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:33.175490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:33.191212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd77871-a66d-4b77-ad81-1d47f6b3fe5c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:33.192650 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd77871-a66d-4b77-ad81-1d47f6b3fe5c service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:33.211103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 668/1341] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:36.049412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eeba42-6152-440f-91c5-559a9c1d473b None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:36.049967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eeba42-6152-440f-91c5-559a9c1d473b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:36.049967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0eeba42-6152-440f-91c5-559a9c1d473b None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:36.051332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 674/1342] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:36.056561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74be937-327f-44f7-a504-6f8b2e9881ed None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:36.057012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74be937-327f-44f7-a504-6f8b2e9881ed None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:36.057186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74be937-327f-44f7-a504-6f8b2e9881ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:36.124314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-a74be937-327f-44f7-a504-6f8b2e9881ed None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:36.125516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a74be937-327f-44f7-a504-6f8b2e9881ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:36.223900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 669/1343] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:36 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:36.530651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d04f8c-63d3-47ac-b782-6ae3997e109f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:36.531002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d04f8c-63d3-47ac-b782-6ae3997e109f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:36.531341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d04f8c-63d3-47ac-b782-6ae3997e109f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:36.532750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 675/1344] 10.222.0.16 () {54 vars in 965 bytes} [Thu Aug 25 05:05:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:36.539158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6b3bc-613a-4afc-a153-5a6c3f6a3012 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:36.539542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6b3bc-613a-4afc-a153-5a6c3f6a3012 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:36.539874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d6b3bc-613a-4afc-a153-5a6c3f6a3012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:36.607312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:36.615574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-94d6b3bc-613a-4afc-a153-5a6c3f6a3012 None None] MFA Rules not processed for user `9220ba3637a542a382ad1d42ff1ff459`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:36.616765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-94d6b3bc-613a-4afc-a153-5a6c3f6a3012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:36.693465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 670/1345] 10.222.0.16 () {58 vars in 1062 bytes} [Thu Aug 25 05:05:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:05:36.703081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d04f8c-63d3-47ac-b782-6ae3997e109f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:36.709792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-24d04f8c-63d3-47ac-b782-6ae3997e109f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:36.811382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d04f8c-63d3-47ac-b782-6ae3997e109f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:36.893146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bb945d-df8b-4db5-a3fe-33f3256648f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:36.893781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bb945d-df8b-4db5-a3fe-33f3256648f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:36.894060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bb945d-df8b-4db5-a3fe-33f3256648f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:36.894335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bb945d-df8b-4db5-a3fe-33f3256648f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:36.895906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-45bb945d-df8b-4db5-a3fe-33f3256648f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:36.939254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:37.007799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bb945d-df8b-4db5-a3fe-33f3256648f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:37.009250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bb945d-df8b-4db5-a3fe-33f3256648f4 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:37.028104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 676/1346] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:36 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:41.932063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77fe57-0215-4ea4-bee1-f267870f267e None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:41.933069 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77fe57-0215-4ea4-bee1-f267870f267e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:41.933182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77fe57-0215-4ea4-bee1-f267870f267e None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:41.934534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 671/1347] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:41.940232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb4ec16-c7d2-4fbf-ba80-41288dba4075 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:41.940824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb4ec16-c7d2-4fbf-ba80-41288dba4075 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:41.940924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb4ec16-c7d2-4fbf-ba80-41288dba4075 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:42.013218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-abb4ec16-c7d2-4fbf-ba80-41288dba4075 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:42.014480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-abb4ec16-c7d2-4fbf-ba80-41288dba4075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:42.112259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:42.115025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 677/1348] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:05:42.400509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb77fe57-0215-4ea4-bee1-f267870f267e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:42.403061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-eb77fe57-0215-4ea4-bee1-f267870f267e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:42.507528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb77fe57-0215-4ea4-bee1-f267870f267e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:42.555260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:42.581840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf86d5-a608-4fb8-bc4b-7f89c901c718 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:42.582446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf86d5-a608-4fb8-bc4b-7f89c901c718 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:42.582729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf86d5-a608-4fb8-bc4b-7f89c901c718 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:42.583003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf86d5-a608-4fb8-bc4b-7f89c901c718 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:42.584602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3cdf86d5-a608-4fb8-bc4b-7f89c901c718 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:42.694056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdf86d5-a608-4fb8-bc4b-7f89c901c718 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:42.695423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdf86d5-a608-4fb8-bc4b-7f89c901c718 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:42.714280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 672/1349] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:05:47.580516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062aa3e5-f425-4883-a653-6be25a7ac6ee None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:47.581044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062aa3e5-f425-4883-a653-6be25a7ac6ee None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:47.581044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062aa3e5-f425-4883-a653-6be25a7ac6ee None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:47.582984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 678/1350] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:05:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:47.588036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd239491-c54e-4b81-ba4a-e8e6b1a97d33 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:47.588587 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd239491-c54e-4b81-ba4a-e8e6b1a97d33 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:47.589711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd239491-c54e-4b81-ba4a-e8e6b1a97d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:47.693975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-cd239491-c54e-4b81-ba4a-e8e6b1a97d33 None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:47.701706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cd239491-c54e-4b81-ba4a-e8e6b1a97d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:47.725323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:47.875901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 673/1351] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:05:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:52.091625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4f4a18-7a73-4e35-93f6-01362f191e5e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:52.092500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4f4a18-7a73-4e35-93f6-01362f191e5e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:52.092928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4f4a18-7a73-4e35-93f6-01362f191e5e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:52.095098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 679/1352] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:05:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:52.099900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20688f90-ea04-4138-b268-314965438c1b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:52.100308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20688f90-ea04-4138-b268-314965438c1b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:52.100474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20688f90-ea04-4138-b268-314965438c1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:52.178719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-20688f90-ea04-4138-b268-314965438c1b None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:52.179924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-20688f90-ea04-4138-b268-314965438c1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:52.264940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:52.303502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 674/1353] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:05:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:53.539012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10521b21-dcc9-417a-92be-b44fffeff023 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:53.539523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10521b21-dcc9-417a-92be-b44fffeff023 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:53.539523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10521b21-dcc9-417a-92be-b44fffeff023 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:53.541006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 680/1354] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:05:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:53.546494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb19e7b-4671-4272-80fd-87958debc45d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:53.546916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb19e7b-4671-4272-80fd-87958debc45d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:53.547049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb19e7b-4671-4272-80fd-87958debc45d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:53.622986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2eb19e7b-4671-4272-80fd-87958debc45d None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:53.624235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2eb19e7b-4671-4272-80fd-87958debc45d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:53.732768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 675/1355] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:05:53 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:57.127112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57961348-def9-48c0-8873-e1651f637855 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:57.127697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57961348-def9-48c0-8873-e1651f637855 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:57.127697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57961348-def9-48c0-8873-e1651f637855 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:57.129088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 681/1356] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:05:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:57.134372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa79dc45-625f-437c-81aa-5237710c34b4 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:57.134802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa79dc45-625f-437c-81aa-5237710c34b4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:57.135512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa79dc45-625f-437c-81aa-5237710c34b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:57.176477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:57.217411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-fa79dc45-625f-437c-81aa-5237710c34b4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:57.218572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-fa79dc45-625f-437c-81aa-5237710c34b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:57.338874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 676/1357] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:05:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:05:57.657980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16657a-1d6f-40e0-a1c6-b63fdd7cd036 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:57.657980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16657a-1d6f-40e0-a1c6-b63fdd7cd036 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:57.658877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16657a-1d6f-40e0-a1c6-b63fdd7cd036 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:57.661017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 682/1358] 10.222.0.16 () {54 vars in 962 bytes} [Thu Aug 25 05:05:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:05:57.670224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d114c-b383-4875-9aea-0ae03ce1679a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:57.671004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d114c-b383-4875-9aea-0ae03ce1679a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:57.671496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d114c-b383-4875-9aea-0ae03ce1679a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:57.756120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-910d114c-b383-4875-9aea-0ae03ce1679a None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:05:57.757742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-910d114c-b383-4875-9aea-0ae03ce1679a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:57.810675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:57.879846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 677/1359] 10.222.0.16 () {58 vars in 1059 bytes} [Thu Aug 25 05:05:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:05:57.887587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16657a-1d6f-40e0-a1c6-b63fdd7cd036 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:05:57.889030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1a16657a-1d6f-40e0-a1c6-b63fdd7cd036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:58.055159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16657a-1d6f-40e0-a1c6-b63fdd7cd036 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:05:58.135471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:05:58.179261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5d0df5-c698-4158-b85e-349fcc1765fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:05:58.180128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d0df5-c698-4158-b85e-349fcc1765fc service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:05:58.180404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d0df5-c698-4158-b85e-349fcc1765fc service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:05:58.180676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5d0df5-c698-4158-b85e-349fcc1765fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:05:58.182416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cc5d0df5-c698-4158-b85e-349fcc1765fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:05:58.278518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5d0df5-c698-4158-b85e-349fcc1765fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:05:58.280273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5d0df5-c698-4158-b85e-349fcc1765fc service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:05:58.299560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 683/1360] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:05:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:00.425151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6000ee4-600e-40ba-b5fb-c6c05cf52907 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:00.425836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6000ee4-600e-40ba-b5fb-c6c05cf52907 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:00.425836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6000ee4-600e-40ba-b5fb-c6c05cf52907 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:00.427302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 678/1361] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:00.433057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b6a03-02ea-4b3f-89d2-fad4be8aa71c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:00.434009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b6a03-02ea-4b3f-89d2-fad4be8aa71c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:00.434443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6b6a03-02ea-4b3f-89d2-fad4be8aa71c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:00.506021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2b6b6a03-02ea-4b3f-89d2-fad4be8aa71c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:00.507309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2b6b6a03-02ea-4b3f-89d2-fad4be8aa71c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:00.547627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:00.613632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 684/1362] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:00.910801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6000ee4-600e-40ba-b5fb-c6c05cf52907 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:00.911504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c6000ee4-600e-40ba-b5fb-c6c05cf52907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:01.073946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6000ee4-600e-40ba-b5fb-c6c05cf52907 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:01.083838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:01.192816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31569ae4-8487-4235-a234-6ccad031ffbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:01.193421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31569ae4-8487-4235-a234-6ccad031ffbb service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:01.193726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31569ae4-8487-4235-a234-6ccad031ffbb service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:01.193996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31569ae4-8487-4235-a234-6ccad031ffbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:01.195589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-31569ae4-8487-4235-a234-6ccad031ffbb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:01.310973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31569ae4-8487-4235-a234-6ccad031ffbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:01.312654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31569ae4-8487-4235-a234-6ccad031ffbb service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:01.345822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 679/1363] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:03.765361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15879f-407d-4150-a1a8-6507f2b97e4f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:03.766287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15879f-407d-4150-a1a8-6507f2b97e4f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:03.766563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15879f-407d-4150-a1a8-6507f2b97e4f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:03.767915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 685/1364] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:03.773119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612fafd0-3465-4b39-b6e5-2c8dd02ba0b2 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:03.773540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612fafd0-3465-4b39-b6e5-2c8dd02ba0b2 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:03.773760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612fafd0-3465-4b39-b6e5-2c8dd02ba0b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:03.789653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:03.842008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-612fafd0-3465-4b39-b6e5-2c8dd02ba0b2 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:03.843225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-612fafd0-3465-4b39-b6e5-2c8dd02ba0b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:03.945936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 680/1365] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:04.239048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2914cbd8-b2a5-436a-b988-f6314e56e119 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:04.239374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2914cbd8-b2a5-436a-b988-f6314e56e119 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:04.239606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2914cbd8-b2a5-436a-b988-f6314e56e119 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:04.241196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 686/1366] 10.222.0.16 () {54 vars in 962 bytes} [Thu Aug 25 05:06:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:04.247126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b39ba46-e7c4-4169-8ac0-8c4b35fad664 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:04.247490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b39ba46-e7c4-4169-8ac0-8c4b35fad664 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:04.247837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b39ba46-e7c4-4169-8ac0-8c4b35fad664 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:04.326841 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4b39ba46-e7c4-4169-8ac0-8c4b35fad664 None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:04.328050 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4b39ba46-e7c4-4169-8ac0-8c4b35fad664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:04.365339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:04.451188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 681/1367] 10.222.0.16 () {58 vars in 1059 bytes} [Thu Aug 25 05:06:04 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:04.459354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2914cbd8-b2a5-436a-b988-f6314e56e119 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:04.460957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2914cbd8-b2a5-436a-b988-f6314e56e119 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:04.616920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2914cbd8-b2a5-436a-b988-f6314e56e119 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:04.639017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:04.737384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd93f8d2-e0a6-487c-a096-976014163d6f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:04.738040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93f8d2-e0a6-487c-a096-976014163d6f service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:04.738317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93f8d2-e0a6-487c-a096-976014163d6f service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:04.738584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93f8d2-e0a6-487c-a096-976014163d6f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:04.740306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cd93f8d2-e0a6-487c-a096-976014163d6f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:04.837106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93f8d2-e0a6-487c-a096-976014163d6f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:04.840047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93f8d2-e0a6-487c-a096-976014163d6f service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:04.859113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 687/1368] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:05.334875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06d58b1-9769-4e72-9f7c-922c38a14e73 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:05.335433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06d58b1-9769-4e72-9f7c-922c38a14e73 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:05.335433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06d58b1-9769-4e72-9f7c-922c38a14e73 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:05.336933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 682/1369] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:05.342528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9218e25e-f49b-4cc9-b1ca-4a04262e32b4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:05.342928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9218e25e-f49b-4cc9-b1ca-4a04262e32b4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:05.343528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9218e25e-f49b-4cc9-b1ca-4a04262e32b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:05.377907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:05.414540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9218e25e-f49b-4cc9-b1ca-4a04262e32b4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:05.416268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9218e25e-f49b-4cc9-b1ca-4a04262e32b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:05.521494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 688/1370] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:06.106310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e02d79-76d6-4236-819f-6b3c75dcafcb None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:06.106667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e02d79-76d6-4236-819f-6b3c75dcafcb None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:06.107009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e02d79-76d6-4236-819f-6b3c75dcafcb None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:06.108573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 683/1371] 10.222.0.16 () {54 vars in 964 bytes} [Thu Aug 25 05:06:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:06.113385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57eba7d-c2b8-4b77-bba1-63880c5de15d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:06.113915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57eba7d-c2b8-4b77-bba1-63880c5de15d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:06.114129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57eba7d-c2b8-4b77-bba1-63880c5de15d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:06.199818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b57eba7d-c2b8-4b77-bba1-63880c5de15d None None] MFA Rules not processed for user `ec312d29dfa846cbace53e8444b8a527`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:06.201101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b57eba7d-c2b8-4b77-bba1-63880c5de15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:06.253098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:06.284649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 689/1372] 10.222.0.16 () {58 vars in 1061 bytes} [Thu Aug 25 05:06:06 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:06.292888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e02d79-76d6-4236-819f-6b3c75dcafcb None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:06.294572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-12e02d79-76d6-4236-819f-6b3c75dcafcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:06.412068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e02d79-76d6-4236-819f-6b3c75dcafcb None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:06.460690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:06.500520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af2a17e-a999-4887-9ae7-2b494df98db0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:06.501129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af2a17e-a999-4887-9ae7-2b494df98db0 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:06.501414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af2a17e-a999-4887-9ae7-2b494df98db0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:06.501773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af2a17e-a999-4887-9ae7-2b494df98db0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:06.503624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1af2a17e-a999-4887-9ae7-2b494df98db0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:06.623606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af2a17e-a999-4887-9ae7-2b494df98db0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:06.625237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af2a17e-a999-4887-9ae7-2b494df98db0 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:06.645982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 684/1373] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:06:06.984401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731efd-2bd3-4f35-a055-fae3624dfabd None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:06.984688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731efd-2bd3-4f35-a055-fae3624dfabd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:06.984963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731efd-2bd3-4f35-a055-fae3624dfabd None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:06.986364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 690/1374] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:06.991590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae323c89-c79c-49db-9c0e-04c4bc5c829d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:06.991875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae323c89-c79c-49db-9c0e-04c4bc5c829d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:06.992150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae323c89-c79c-49db-9c0e-04c4bc5c829d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:07.058891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ae323c89-c79c-49db-9c0e-04c4bc5c829d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:07.060146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ae323c89-c79c-49db-9c0e-04c4bc5c829d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:07.080690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:07.159987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 685/1375] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:07.453183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a731efd-2bd3-4f35-a055-fae3624dfabd None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:07.454995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6a731efd-2bd3-4f35-a055-fae3624dfabd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:07.601336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a731efd-2bd3-4f35-a055-fae3624dfabd None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:07.653984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:07.715601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02de6892-933f-4edf-a6a3-35d3ca23fb3e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:07.716463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de6892-933f-4edf-a6a3-35d3ca23fb3e service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:07.716742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de6892-933f-4edf-a6a3-35d3ca23fb3e service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:07.717014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02de6892-933f-4edf-a6a3-35d3ca23fb3e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:07.718645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-02de6892-933f-4edf-a6a3-35d3ca23fb3e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:07.808537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02de6892-933f-4edf-a6a3-35d3ca23fb3e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:07.810163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02de6892-933f-4edf-a6a3-35d3ca23fb3e service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:07.828176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 691/1376] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:09.868248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a7633-9ef6-45c6-9f57-88e00c0447be None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:09.868821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a7633-9ef6-45c6-9f57-88e00c0447be None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:09.868821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a7633-9ef6-45c6-9f57-88e00c0447be None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:09.870546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 686/1377] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:09.882242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49541736-688d-4939-b31d-3b90a41f3c85 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:09.882734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49541736-688d-4939-b31d-3b90a41f3c85 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:09.882853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49541736-688d-4939-b31d-3b90a41f3c85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:09.948959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-49541736-688d-4939-b31d-3b90a41f3c85 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:09.950110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-49541736-688d-4939-b31d-3b90a41f3c85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:09.958906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:10.049678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 692/1378] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:10.360711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676a7633-9ef6-45c6-9f57-88e00c0447be None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:10.362400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-676a7633-9ef6-45c6-9f57-88e00c0447be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:10.530977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676a7633-9ef6-45c6-9f57-88e00c0447be None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:10.535091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:10.647784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e366a3-cf1d-40a1-830b-815f6cd99afc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:10.648506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e366a3-cf1d-40a1-830b-815f6cd99afc service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:10.648880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e366a3-cf1d-40a1-830b-815f6cd99afc service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:10.649336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e366a3-cf1d-40a1-830b-815f6cd99afc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:10.651610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-84e366a3-cf1d-40a1-830b-815f6cd99afc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:10.750177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e366a3-cf1d-40a1-830b-815f6cd99afc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:10.751865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e366a3-cf1d-40a1-830b-815f6cd99afc service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:10.772078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 687/1379] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:12.822637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d89bc05-a05e-471b-983b-8576c965d45f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:12.823203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d89bc05-a05e-471b-983b-8576c965d45f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:12.823203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d89bc05-a05e-471b-983b-8576c965d45f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:12.824594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 693/1380] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:12.829524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0932bed3-5384-4bf2-8277-4c05ebe72208 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:12.829947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0932bed3-5384-4bf2-8277-4c05ebe72208 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:12.830106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0932bed3-5384-4bf2-8277-4c05ebe72208 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:12.842065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:12.902760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-0932bed3-5384-4bf2-8277-4c05ebe72208 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:12.904060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0932bed3-5384-4bf2-8277-4c05ebe72208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:13.008497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 688/1381] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:13.304175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d89bc05-a05e-471b-983b-8576c965d45f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:13.305800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7d89bc05-a05e-471b-983b-8576c965d45f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:13.461853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:13.473203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d89bc05-a05e-471b-983b-8576c965d45f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:13.601882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55d5de5-dbef-427d-8a51-c569ed3ae3bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:13.602498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55d5de5-dbef-427d-8a51-c569ed3ae3bc service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:13.602776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55d5de5-dbef-427d-8a51-c569ed3ae3bc service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:13.603118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55d5de5-dbef-427d-8a51-c569ed3ae3bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:13.605519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a55d5de5-dbef-427d-8a51-c569ed3ae3bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:13.704610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55d5de5-dbef-427d-8a51-c569ed3ae3bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:13.706414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55d5de5-dbef-427d-8a51-c569ed3ae3bc service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:13.719174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:13.728466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 694/1382] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:16.358332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6264c-5718-4e9e-bb68-5a51598b3d92 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:16.358859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6264c-5718-4e9e-bb68-5a51598b3d92 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:16.358957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6264c-5718-4e9e-bb68-5a51598b3d92 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:16.360254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 689/1383] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:16.365605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5275d917-2156-4701-971c-632f549230e9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:16.366259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5275d917-2156-4701-971c-632f549230e9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:16.366539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5275d917-2156-4701-971c-632f549230e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:16.438167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5275d917-2156-4701-971c-632f549230e9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:16.439392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5275d917-2156-4701-971c-632f549230e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:16.539235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 695/1384] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:16.827541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6264c-5718-4e9e-bb68-5a51598b3d92 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:16.829039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cea6264c-5718-4e9e-bb68-5a51598b3d92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:16.925096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:16.987172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6264c-5718-4e9e-bb68-5a51598b3d92 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:17.222524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df44effc-0736-4c6a-b975-1b1b33ddd5f8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:17.223294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df44effc-0736-4c6a-b975-1b1b33ddd5f8 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:17.223431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df44effc-0736-4c6a-b975-1b1b33ddd5f8 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:17.223692 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df44effc-0736-4c6a-b975-1b1b33ddd5f8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:17.225414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-df44effc-0736-4c6a-b975-1b1b33ddd5f8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:17.276338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:17.316681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df44effc-0736-4c6a-b975-1b1b33ddd5f8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:17.318320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df44effc-0736-4c6a-b975-1b1b33ddd5f8 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:17.336102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 690/1385] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 509 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:19.241256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ec8652-2cfd-440f-8d7c-c7723223366a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:19.241798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ec8652-2cfd-440f-8d7c-c7723223366a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:19.241941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ec8652-2cfd-440f-8d7c-c7723223366a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:19.243462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 696/1386] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:19.249571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5681fe-842c-45e1-9a64-44c73e650217 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:19.250002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5681fe-842c-45e1-9a64-44c73e650217 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:19.250146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5681fe-842c-45e1-9a64-44c73e650217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:19.325393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-7a5681fe-842c-45e1-9a64-44c73e650217 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:19.327030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7a5681fe-842c-45e1-9a64-44c73e650217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:19.433215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 691/1387] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:19.725080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ec8652-2cfd-440f-8d7c-c7723223366a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:19.727235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c4ec8652-2cfd-440f-8d7c-c7723223366a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:19.812339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:19.881921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ec8652-2cfd-440f-8d7c-c7723223366a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:19.997989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0127e2df-f8dd-408e-ba57-e6b14c76a9cd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:19.998522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0127e2df-f8dd-408e-ba57-e6b14c76a9cd service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:19.998800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0127e2df-f8dd-408e-ba57-e6b14c76a9cd service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:19.999072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0127e2df-f8dd-408e-ba57-e6b14c76a9cd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:20.000659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0127e2df-f8dd-408e-ba57-e6b14c76a9cd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:20.039262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:20.092696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0127e2df-f8dd-408e-ba57-e6b14c76a9cd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:20.094429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0127e2df-f8dd-408e-ba57-e6b14c76a9cd service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:20.113765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 697/1388] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:21.916969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff519a-c5d5-450b-87aa-39c2f0492dd0 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:21.917529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff519a-c5d5-450b-87aa-39c2f0492dd0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:21.917529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff519a-c5d5-450b-87aa-39c2f0492dd0 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:21.918902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 692/1389] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:21.924505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e77444-0a51-4f65-916f-23e328d9bd3c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:21.924958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e77444-0a51-4f65-916f-23e328d9bd3c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:21.925063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e77444-0a51-4f65-916f-23e328d9bd3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:21.991866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-54e77444-0a51-4f65-916f-23e328d9bd3c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:21.993806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-54e77444-0a51-4f65-916f-23e328d9bd3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:22.092005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 698/1390] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:22.168041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993b949-7101-4d1f-994e-dadb9de8f42c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:22.168236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993b949-7101-4d1f-994e-dadb9de8f42c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:22.168522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b993b949-7101-4d1f-994e-dadb9de8f42c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:22.169908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 693/1391] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:22.175285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ebb1bd-ef76-4936-9fd9-5c1e7c4e2ae0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:22.175570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ebb1bd-ef76-4936-9fd9-5c1e7c4e2ae0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:22.175843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ebb1bd-ef76-4936-9fd9-5c1e7c4e2ae0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:22.192061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:22.245168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-43ebb1bd-ef76-4936-9fd9-5c1e7c4e2ae0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:22.246514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-43ebb1bd-ef76-4936-9fd9-5c1e7c4e2ae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:22.348493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 699/1392] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:22.392586 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b762c56-9f02-49fd-b74a-979d8c063b9b None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:22.392875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b762c56-9f02-49fd-b74a-979d8c063b9b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:22.393149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b762c56-9f02-49fd-b74a-979d8c063b9b None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:22.394501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 694/1393] 10.222.0.16 () {54 vars in 967 bytes} [Thu Aug 25 05:06:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:22.400434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e955771a-775b-4dab-b7ea-0a792b08eeaa None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:22.400955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e955771a-775b-4dab-b7ea-0a792b08eeaa None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:22.401406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e955771a-775b-4dab-b7ea-0a792b08eeaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:22.476652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e955771a-775b-4dab-b7ea-0a792b08eeaa None None] MFA Rules not processed for user `a6cc8f34f9674a94ba23a7157e44a6f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:22.478401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e955771a-775b-4dab-b7ea-0a792b08eeaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:22.512360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:22.554810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 700/1394] 10.222.0.16 () {58 vars in 1064 bytes} [Thu Aug 25 05:06:22 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:22.562499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b762c56-9f02-49fd-b74a-979d8c063b9b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:22.564011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0b762c56-9f02-49fd-b74a-979d8c063b9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:22.595589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:22.640162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e955771a-775b-4dab-b7ea-0a792b08eeaa None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:22.641440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e955771a-775b-4dab-b7ea-0a792b08eeaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:22.669718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b762c56-9f02-49fd-b74a-979d8c063b9b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:22.746814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585ca1f4-2eeb-4d3b-9a5c-5162dab6e242 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6cc8f34f9674a94ba23a7157e44a6f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:22.747355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ca1f4-2eeb-4d3b-9a5c-5162dab6e242 service glance] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:22.747635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ca1f4-2eeb-4d3b-9a5c-5162dab6e242 service glance] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:22.747909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ca1f4-2eeb-4d3b-9a5c-5162dab6e242 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:22.749501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-585ca1f4-2eeb-4d3b-9a5c-5162dab6e242 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:22.764483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e955771a-775b-4dab-b7ea-0a792b08eeaa None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:22.828325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:22.828678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:22.861403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585ca1f4-2eeb-4d3b-9a5c-5162dab6e242 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:22.862863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585ca1f4-2eeb-4d3b-9a5c-5162dab6e242 service glance] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:22.880952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 695/1395] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:22.881353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2db8b2c-0977-427d-be0a-4bba891e4e2a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:22.882147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2db8b2c-0977-427d-be0a-4bba891e4e2a service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:22.882432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2db8b2c-0977-427d-be0a-4bba891e4e2a service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:22.882704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2db8b2c-0977-427d-be0a-4bba891e4e2a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:22.884313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d2db8b2c-0977-427d-be0a-4bba891e4e2a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:22.972852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2db8b2c-0977-427d-be0a-4bba891e4e2a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:22.974466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2db8b2c-0977-427d-be0a-4bba891e4e2a service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:22.992698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 701/1396] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:06:23.745355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcecfdc-99ad-4777-b6b1-ff84f25ac794 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:23.745842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcecfdc-99ad-4777-b6b1-ff84f25ac794 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:23.746008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcecfdc-99ad-4777-b6b1-ff84f25ac794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:23.812712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-0dcecfdc-99ad-4777-b6b1-ff84f25ac794 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:23.814246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0dcecfdc-99ad-4777-b6b1-ff84f25ac794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:23.900681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 696/1397] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:06:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:23.942849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015cecaf-d60d-4d8c-9c5c-af8a5b8e6297 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:23.943214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015cecaf-d60d-4d8c-9c5c-af8a5b8e6297 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:23.943564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015cecaf-d60d-4d8c-9c5c-af8a5b8e6297 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:23.945080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 702/1398] 10.222.0.16 () {54 vars in 943 bytes} [Thu Aug 25 05:06:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:23.951390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe93fd2f-39d6-4422-922e-35aca0822e78 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:23.951679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe93fd2f-39d6-4422-922e-35aca0822e78 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:23.951951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe93fd2f-39d6-4422-922e-35aca0822e78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:24.008391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:24.026945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-fe93fd2f-39d6-4422-922e-35aca0822e78 None None] MFA Rules not processed for user `c77473684c58457aa53f73a9c30319d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:24.030401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-fe93fd2f-39d6-4422-922e-35aca0822e78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:24.134722 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 697/1399] 10.222.0.16 () {58 vars in 1040 bytes} [Thu Aug 25 05:06:23 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:24.144589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015cecaf-d60d-4d8c-9c5c-af8a5b8e6297 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:24.146172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-015cecaf-d60d-4d8c-9c5c-af8a5b8e6297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:24.229202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:24.289557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015cecaf-d60d-4d8c-9c5c-af8a5b8e6297 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:24.393163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8ac733-1181-4fbc-8c81-23d4376920a9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:24.393804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ac733-1181-4fbc-8c81-23d4376920a9 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:24.394085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ac733-1181-4fbc-8c81-23d4376920a9 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:24.394359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ac733-1181-4fbc-8c81-23d4376920a9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:24.395985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ec8ac733-1181-4fbc-8c81-23d4376920a9 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:24.455476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:24.477987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8ac733-1181-4fbc-8c81-23d4376920a9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:24.479582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8ac733-1181-4fbc-8c81-23d4376920a9 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:24.488806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 703/1400] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:06:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:06:25.056724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f809b37c-fcfd-4720-a800-8ff9c2a10eb3 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:25.057275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f809b37c-fcfd-4720-a800-8ff9c2a10eb3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:25.057275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f809b37c-fcfd-4720-a800-8ff9c2a10eb3 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:25.058620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 698/1401] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:25.066841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef7190-0007-463f-9601-55ab1725c842 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:25.067133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef7190-0007-463f-9601-55ab1725c842 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:25.067413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aef7190-0007-463f-9601-55ab1725c842 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:25.136330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1aef7190-0007-463f-9601-55ab1725c842 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:25.137609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1aef7190-0007-463f-9601-55ab1725c842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:25.244266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 704/1402] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:25.539016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f809b37c-fcfd-4720-a800-8ff9c2a10eb3 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:25.541006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f809b37c-fcfd-4720-a800-8ff9c2a10eb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:25.656920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:25.717100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f809b37c-fcfd-4720-a800-8ff9c2a10eb3 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:25.850439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5691ae7f-a74b-4edd-8b34-9029b132a3e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:25.850439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691ae7f-a74b-4edd-8b34-9029b132a3e4 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:25.850439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691ae7f-a74b-4edd-8b34-9029b132a3e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:25.850439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5691ae7f-a74b-4edd-8b34-9029b132a3e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:25.851031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5691ae7f-a74b-4edd-8b34-9029b132a3e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:25.934319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:25.957122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5691ae7f-a74b-4edd-8b34-9029b132a3e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:25.958965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5691ae7f-a74b-4edd-8b34-9029b132a3e4 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:25.989343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 699/1403] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:28.562936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ae0968-bf42-4629-84a1-d6b5cb41e60e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:28.563511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ae0968-bf42-4629-84a1-d6b5cb41e60e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:28.563511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ae0968-bf42-4629-84a1-d6b5cb41e60e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:28.564875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 705/1404] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:28.570556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d58b64d-517d-44f1-a486-3f49012f49e6 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:28.571032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d58b64d-517d-44f1-a486-3f49012f49e6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:28.571200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d58b64d-517d-44f1-a486-3f49012f49e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:28.654565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-0d58b64d-517d-44f1-a486-3f49012f49e6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:28.655787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0d58b64d-517d-44f1-a486-3f49012f49e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:28.768092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 700/1405] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:29.056125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ae0968-bf42-4629-84a1-d6b5cb41e60e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:29.057677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-42ae0968-bf42-4629-84a1-d6b5cb41e60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:29.119794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:29.231446 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ae0968-bf42-4629-84a1-d6b5cb41e60e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:29.364360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab7750f-42ed-44d7-ba5d-a2163c27e1a4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:29.365958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab7750f-42ed-44d7-ba5d-a2163c27e1a4 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:29.366257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab7750f-42ed-44d7-ba5d-a2163c27e1a4 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:29.366534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab7750f-42ed-44d7-ba5d-a2163c27e1a4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:29.368701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8ab7750f-42ed-44d7-ba5d-a2163c27e1a4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:29.381983 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:29.472141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab7750f-42ed-44d7-ba5d-a2163c27e1a4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:29.473774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab7750f-42ed-44d7-ba5d-a2163c27e1a4 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:29.496909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 706/1406] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:31.618097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be7c0fa-c5b5-42e4-94fd-6d5abb81e150 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:31.618097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be7c0fa-c5b5-42e4-94fd-6d5abb81e150 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:31.618097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be7c0fa-c5b5-42e4-94fd-6d5abb81e150 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:31.619865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 701/1407] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:06:31 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:31.625081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c31b59a-1c77-4fa7-834d-795b727bb60e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:31.625674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c31b59a-1c77-4fa7-834d-795b727bb60e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:31.625674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c31b59a-1c77-4fa7-834d-795b727bb60e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:31.709991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3c31b59a-1c77-4fa7-834d-795b727bb60e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:31.733613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3c31b59a-1c77-4fa7-834d-795b727bb60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:31.836771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:31.848510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 707/1408] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:06:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:32.145885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be7c0fa-c5b5-42e4-94fd-6d5abb81e150 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:32.147813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9be7c0fa-c5b5-42e4-94fd-6d5abb81e150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:32.202251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:32.316233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be7c0fa-c5b5-42e4-94fd-6d5abb81e150 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:32.452346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b422e68-9dca-4b2b-a14c-2489baca6784 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:32.452925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b422e68-9dca-4b2b-a14c-2489baca6784 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:32.454307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b422e68-9dca-4b2b-a14c-2489baca6784 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:32.454596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b422e68-9dca-4b2b-a14c-2489baca6784 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:32.456311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3b422e68-9dca-4b2b-a14c-2489baca6784 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:32.555027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b422e68-9dca-4b2b-a14c-2489baca6784 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:32.557788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b422e68-9dca-4b2b-a14c-2489baca6784 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:32.563011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:32.580038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 702/1409] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:42.505313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:42.598534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=42 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:48.414730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6298de0-1d34-47c2-8208-7b073d32af23 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:48.415262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6298de0-1d34-47c2-8208-7b073d32af23 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:48.415262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6298de0-1d34-47c2-8208-7b073d32af23 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:48.416615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 708/1410] 10.222.0.16 () {54 vars in 918 bytes} [Thu Aug 25 05:06:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:48.421567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee1dba-5f5b-42f0-9e06-7462ee2d0179 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:48.421982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee1dba-5f5b-42f0-9e06-7462ee2d0179 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:48.422155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ee1dba-5f5b-42f0-9e06-7462ee2d0179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:48.496131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f6ee1dba-5f5b-42f0-9e06-7462ee2d0179 None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:48.497243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f6ee1dba-5f5b-42f0-9e06-7462ee2d0179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:48.600311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 703/1411] 10.222.0.16 () {58 vars in 1015 bytes} [Thu Aug 25 05:06:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:48.614460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b4afc3-769b-4da1-b7fc-2a2cd3a4eb39 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:48.614745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b4afc3-769b-4da1-b7fc-2a2cd3a4eb39 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:48.615050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b4afc3-769b-4da1-b7fc-2a2cd3a4eb39 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:48.616382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 709/1412] 10.222.0.16 () {54 vars in 918 bytes} [Thu Aug 25 05:06:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:48.621539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303aa7a8-25cf-48a1-b1f4-6f3c217f4c8b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:48.621852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303aa7a8-25cf-48a1-b1f4-6f3c217f4c8b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:48.622125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303aa7a8-25cf-48a1-b1f4-6f3c217f4c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:48.694101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:48.704120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-303aa7a8-25cf-48a1-b1f4-6f3c217f4c8b None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:48.704120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-303aa7a8-25cf-48a1-b1f4-6f3c217f4c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:48.811877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 704/1413] 10.222.0.16 () {58 vars in 1015 bytes} [Thu Aug 25 05:06:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:51.260979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1bd50a-4f88-40b9-930e-99f76d17a948 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:51.261426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1bd50a-4f88-40b9-930e-99f76d17a948 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:51.261543 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1bd50a-4f88-40b9-930e-99f76d17a948 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:51.262888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 710/1414] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:06:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:51.268345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd0bc49-95d3-453b-819b-9a056d21ae3d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:51.268766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd0bc49-95d3-453b-819b-9a056d21ae3d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:51.268873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd0bc49-95d3-453b-819b-9a056d21ae3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:51.333635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1dd0bc49-95d3-453b-819b-9a056d21ae3d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:51.334876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1dd0bc49-95d3-453b-819b-9a056d21ae3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:51.389730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:51.431610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 705/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:06:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:51.722969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1bd50a-4f88-40b9-930e-99f76d17a948 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:51.724471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9c1bd50a-4f88-40b9-930e-99f76d17a948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:51.867185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1bd50a-4f88-40b9-930e-99f76d17a948 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:51.937090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:51.977047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00daf0f9-ae12-4cec-aea0-929fc772c0bb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:51.977580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00daf0f9-ae12-4cec-aea0-929fc772c0bb service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:51.977886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00daf0f9-ae12-4cec-aea0-929fc772c0bb service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:51.978156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00daf0f9-ae12-4cec-aea0-929fc772c0bb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:51.979719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-00daf0f9-ae12-4cec-aea0-929fc772c0bb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:52.073954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00daf0f9-ae12-4cec-aea0-929fc772c0bb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:52.075529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00daf0f9-ae12-4cec-aea0-929fc772c0bb service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:52.093679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 711/1416] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:52.984789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a53074f-60b7-4c0e-a489-a9c80b5ff748 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:52.985360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a53074f-60b7-4c0e-a489-a9c80b5ff748 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:52.985360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a53074f-60b7-4c0e-a489-a9c80b5ff748 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:52.986730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 706/1417] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:06:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:52.992879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c24714f-8ed5-46e6-b3dc-963d40e61081 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:52.992879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c24714f-8ed5-46e6-b3dc-963d40e61081 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:52.992879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c24714f-8ed5-46e6-b3dc-963d40e61081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:53.066766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0c24714f-8ed5-46e6-b3dc-963d40e61081 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:53.067950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0c24714f-8ed5-46e6-b3dc-963d40e61081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:53.095264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:53.162940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 712/1418] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:06:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:53.322734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a53074f-60b7-4c0e-a489-a9c80b5ff748 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:53.324173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9a53074f-60b7-4c0e-a489-a9c80b5ff748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:53.467824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a53074f-60b7-4c0e-a489-a9c80b5ff748 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:53.508120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:53.581074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31f3563-79d5-48b9-9b31-f10f71197039 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:53.581616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f3563-79d5-48b9-9b31-f10f71197039 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:53.581900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f3563-79d5-48b9-9b31-f10f71197039 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:53.582163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31f3563-79d5-48b9-9b31-f10f71197039 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:53.583745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e31f3563-79d5-48b9-9b31-f10f71197039 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:53.673960 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f3563-79d5-48b9-9b31-f10f71197039 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:53.675526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31f3563-79d5-48b9-9b31-f10f71197039 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:53.693551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 707/1419] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:06:54.572534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050fd1f-9597-4a4b-9cce-293ec92ad388 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:54.573065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050fd1f-9597-4a4b-9cce-293ec92ad388 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:54.573065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050fd1f-9597-4a4b-9cce-293ec92ad388 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:54.574372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 713/1420] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:54.579238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489baa29-c6c0-40df-a54b-e592c7e01f7b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:54.579657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489baa29-c6c0-40df-a54b-e592c7e01f7b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:54.579799 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489baa29-c6c0-40df-a54b-e592c7e01f7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:54.633705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:54.654835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-489baa29-c6c0-40df-a54b-e592c7e01f7b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:54.656063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-489baa29-c6c0-40df-a54b-e592c7e01f7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:54.752764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 708/1421] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:06:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:54.903251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d050fd1f-9597-4a4b-9cce-293ec92ad388 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:54.904869 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d050fd1f-9597-4a4b-9cce-293ec92ad388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:55.051405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d050fd1f-9597-4a4b-9cce-293ec92ad388 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:55.063356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:55.163135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219b2085-2872-4c01-b9f9-24b41efcdcfa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:55.163694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219b2085-2872-4c01-b9f9-24b41efcdcfa service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:55.163967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219b2085-2872-4c01-b9f9-24b41efcdcfa service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:55.164234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219b2085-2872-4c01-b9f9-24b41efcdcfa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:55.165838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-219b2085-2872-4c01-b9f9-24b41efcdcfa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:55.253137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219b2085-2872-4c01-b9f9-24b41efcdcfa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:55.254752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219b2085-2872-4c01-b9f9-24b41efcdcfa service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:55.272642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 714/1422] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:56.131069 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c8bd1-522b-47f9-ba46-da8e9f7611d3 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:56.131636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c8bd1-522b-47f9-ba46-da8e9f7611d3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:56.131636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c8bd1-522b-47f9-ba46-da8e9f7611d3 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:56.133031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 709/1423] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:06:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:56.137875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ff7cd0-2444-437c-9afb-e90ee489c240 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:56.138342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ff7cd0-2444-437c-9afb-e90ee489c240 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:56.138447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ff7cd0-2444-437c-9afb-e90ee489c240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:56.158146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:56.217519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f0ff7cd0-2444-437c-9afb-e90ee489c240 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:56.219352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f0ff7cd0-2444-437c-9afb-e90ee489c240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:56.313246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 715/1424] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:06:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:56.468918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c8bd1-522b-47f9-ba46-da8e9f7611d3 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:56.470480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-223c8bd1-522b-47f9-ba46-da8e9f7611d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:56.579088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:56.615237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c8bd1-522b-47f9-ba46-da8e9f7611d3 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:56.728200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54dea089-be55-41f7-b03f-233a709c88c7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:56.728740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dea089-be55-41f7-b03f-233a709c88c7 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:56.729020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dea089-be55-41f7-b03f-233a709c88c7 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:56.729293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dea089-be55-41f7-b03f-233a709c88c7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:56.730917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-54dea089-be55-41f7-b03f-233a709c88c7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:56.805489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:56.818223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dea089-be55-41f7-b03f-233a709c88c7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:56.820451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dea089-be55-41f7-b03f-233a709c88c7 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:56.838383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 710/1425] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:57.684285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d81484-d2bb-4126-b539-32f18571f6c3 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:57.684824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d81484-d2bb-4126-b539-32f18571f6c3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:57.684824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d81484-d2bb-4126-b539-32f18571f6c3 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:57.686208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 716/1426] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:06:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:57.691214 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b10cdb-f901-4ede-87fa-cce6e8e331f2 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:57.691627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b10cdb-f901-4ede-87fa-cce6e8e331f2 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:57.691763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b10cdb-f901-4ede-87fa-cce6e8e331f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:57.767335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-04b10cdb-f901-4ede-87fa-cce6e8e331f2 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:57.768518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-04b10cdb-f901-4ede-87fa-cce6e8e331f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:57.861743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 711/1427] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:06:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:58.011500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d81484-d2bb-4126-b539-32f18571f6c3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:58.013089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c0d81484-d2bb-4126-b539-32f18571f6c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:58.097662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:58.160280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d81484-d2bb-4126-b539-32f18571f6c3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:58.271084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb2e40a-ccff-473f-a546-a4b6b62e7306 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:58.271682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2e40a-ccff-473f-a546-a4b6b62e7306 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:58.271955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2e40a-ccff-473f-a546-a4b6b62e7306 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:58.272220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2e40a-ccff-473f-a546-a4b6b62e7306 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:58.274346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4bb2e40a-ccff-473f-a546-a4b6b62e7306 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:58.320905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:58.362562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb2e40a-ccff-473f-a546-a4b6b62e7306 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:58.364157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb2e40a-ccff-473f-a546-a4b6b62e7306 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:58.381895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 717/1428] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:06:59.234435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d17e18a-239c-41dc-9d72-437fb2017a6f None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:59.234853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d17e18a-239c-41dc-9d72-437fb2017a6f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:59.235050 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d17e18a-239c-41dc-9d72-437fb2017a6f None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:59.236346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 712/1429] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:06:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:06:59.242117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54c96a2-a3e4-4c96-84f3-37fdb2371763 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:59.242547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54c96a2-a3e4-4c96-84f3-37fdb2371763 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:59.242651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54c96a2-a3e4-4c96-84f3-37fdb2371763 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:59.319143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c54c96a2-a3e4-4c96-84f3-37fdb2371763 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:06:59.320848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c54c96a2-a3e4-4c96-84f3-37fdb2371763 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:59.417626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 718/1430] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:06:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:06:59.568347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d17e18a-239c-41dc-9d72-437fb2017a6f None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:06:59.570532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2d17e18a-239c-41dc-9d72-437fb2017a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:59.619477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:59.716669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d17e18a-239c-41dc-9d72-437fb2017a6f None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:06:59.827004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5032206b-026e-4afb-bf3e-441895bc7294 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:06:59.827537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032206b-026e-4afb-bf3e-441895bc7294 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:06:59.827816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032206b-026e-4afb-bf3e-441895bc7294 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:06:59.828084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032206b-026e-4afb-bf3e-441895bc7294 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:06:59.829730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5032206b-026e-4afb-bf3e-441895bc7294 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:06:59.836933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:06:59.918503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5032206b-026e-4afb-bf3e-441895bc7294 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:06:59.920074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5032206b-026e-4afb-bf3e-441895bc7294 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:06:59.941140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 713/1431] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:06:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:00.786712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085ec94-3eea-472a-b9c4-2c4632b6c429 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:00.787287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085ec94-3eea-472a-b9c4-2c4632b6c429 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:00.787287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d085ec94-3eea-472a-b9c4-2c4632b6c429 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:00.788626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 719/1432] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:07:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:00.793477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df2cb1-73ab-4dd4-9025-e67b639941bc None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:00.793893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df2cb1-73ab-4dd4-9025-e67b639941bc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:00.794117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df2cb1-73ab-4dd4-9025-e67b639941bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:00.869394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-12df2cb1-73ab-4dd4-9025-e67b639941bc None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:00.870645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-12df2cb1-73ab-4dd4-9025-e67b639941bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:00.937041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:00.965452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 714/1433] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:07:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:01.115144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d085ec94-3eea-472a-b9c4-2c4632b6c429 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:01.117187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d085ec94-3eea-472a-b9c4-2c4632b6c429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:01.134988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:01.264224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d085ec94-3eea-472a-b9c4-2c4632b6c429 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:01.352941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:01.378252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74251a35-6955-45a7-84e1-a83eb63d9791 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:01.378915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74251a35-6955-45a7-84e1-a83eb63d9791 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:01.379189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74251a35-6955-45a7-84e1-a83eb63d9791 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:01.379454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74251a35-6955-45a7-84e1-a83eb63d9791 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:01.381061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-74251a35-6955-45a7-84e1-a83eb63d9791 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:01.473079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74251a35-6955-45a7-84e1-a83eb63d9791 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:01.474693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74251a35-6955-45a7-84e1-a83eb63d9791 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:01.492884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 720/1434] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:02.335524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49173e8f-b326-486c-86d9-a6318e3da901 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:02.336085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49173e8f-b326-486c-86d9-a6318e3da901 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:02.336085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49173e8f-b326-486c-86d9-a6318e3da901 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:02.337406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 715/1435] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:07:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:02.342692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cd2e9c-da57-49e5-820a-62bd812a1053 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:02.343114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cd2e9c-da57-49e5-820a-62bd812a1053 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:02.343240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cd2e9c-da57-49e5-820a-62bd812a1053 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:02.418029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-70cd2e9c-da57-49e5-820a-62bd812a1053 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:02.419214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-70cd2e9c-da57-49e5-820a-62bd812a1053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:02.459464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:02.521864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 721/1436] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:07:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:02.671147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49173e8f-b326-486c-86d9-a6318e3da901 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:02.672675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-49173e8f-b326-486c-86d9-a6318e3da901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:02.816308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49173e8f-b326-486c-86d9-a6318e3da901 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:02.869046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:02.925728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e641be0-d4c3-4fe2-a56f-e9b138fd381a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:02.926333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e641be0-d4c3-4fe2-a56f-e9b138fd381a service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:02.926613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e641be0-d4c3-4fe2-a56f-e9b138fd381a service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:02.926884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e641be0-d4c3-4fe2-a56f-e9b138fd381a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:02.928460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0e641be0-d4c3-4fe2-a56f-e9b138fd381a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:03.015992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e641be0-d4c3-4fe2-a56f-e9b138fd381a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:03.017579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e641be0-d4c3-4fe2-a56f-e9b138fd381a service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:03.035203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 716/1437] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:03.895117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4d8e9a-46cd-4619-85e3-962f335372d6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:03.895654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4d8e9a-46cd-4619-85e3-962f335372d6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:03.895654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4d8e9a-46cd-4619-85e3-962f335372d6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:03.897058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 722/1438] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:07:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:03.901849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563484f-ad1b-4e89-8226-314ce2ea44dc None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:03.902249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563484f-ad1b-4e89-8226-314ce2ea44dc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:03.902395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563484f-ad1b-4e89-8226-314ce2ea44dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:03.976640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3563484f-ad1b-4e89-8226-314ce2ea44dc None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:03.984169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3563484f-ad1b-4e89-8226-314ce2ea44dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:03.984169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:04.074581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 717/1439] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:07:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:04.223664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4d8e9a-46cd-4619-85e3-962f335372d6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:04.225218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8a4d8e9a-46cd-4619-85e3-962f335372d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:04.369094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4d8e9a-46cd-4619-85e3-962f335372d6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:04.395254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:04.481566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58b9623-1b85-423d-83ea-6d7b7f3c2f2c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:04.482140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b9623-1b85-423d-83ea-6d7b7f3c2f2c service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:04.482402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b9623-1b85-423d-83ea-6d7b7f3c2f2c service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:04.482679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58b9623-1b85-423d-83ea-6d7b7f3c2f2c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:04.484262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b58b9623-1b85-423d-83ea-6d7b7f3c2f2c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:04.573621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58b9623-1b85-423d-83ea-6d7b7f3c2f2c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:04.575241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58b9623-1b85-423d-83ea-6d7b7f3c2f2c service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:04.593102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 723/1440] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:06.385669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4224b123-72c3-487a-b777-15a867001c0c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:06.386231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4224b123-72c3-487a-b777-15a867001c0c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:06.386355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4224b123-72c3-487a-b777-15a867001c0c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:06.387616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 718/1441] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:06.393002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee7676f-4df6-4b2d-9fbd-4ec4eff3836b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:06.393413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee7676f-4df6-4b2d-9fbd-4ec4eff3836b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:06.393564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee7676f-4df6-4b2d-9fbd-4ec4eff3836b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:06.435019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:06.459379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-eee7676f-4df6-4b2d-9fbd-4ec4eff3836b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:06.460635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-eee7676f-4df6-4b2d-9fbd-4ec4eff3836b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:06.556949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 724/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:06.842969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4224b123-72c3-487a-b777-15a867001c0c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:06.844531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4224b123-72c3-487a-b777-15a867001c0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:06.947661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4224b123-72c3-487a-b777-15a867001c0c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:06.979562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:07.020375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bff5f3e-0a30-41b6-b10c-8e913acf1069 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:07.020982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bff5f3e-0a30-41b6-b10c-8e913acf1069 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:07.021266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bff5f3e-0a30-41b6-b10c-8e913acf1069 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:07.021534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bff5f3e-0a30-41b6-b10c-8e913acf1069 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:07.023768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5bff5f3e-0a30-41b6-b10c-8e913acf1069 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:07.128580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bff5f3e-0a30-41b6-b10c-8e913acf1069 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:07.130040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bff5f3e-0a30-41b6-b10c-8e913acf1069 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:07.149005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 719/1443] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:09.219150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dc1657-ac13-459c-9197-e2a1557939b3 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:09.219714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dc1657-ac13-459c-9197-e2a1557939b3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:09.219714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dc1657-ac13-459c-9197-e2a1557939b3 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:09.221232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 725/1444] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:09.226597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b719328a-9614-4987-b230-dc7272d7276e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:09.227018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b719328a-9614-4987-b230-dc7272d7276e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:09.227018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b719328a-9614-4987-b230-dc7272d7276e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:09.291437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-b719328a-9614-4987-b230-dc7272d7276e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:09.292562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b719328a-9614-4987-b230-dc7272d7276e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:09.305000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:09.388330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 720/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:09.672606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dc1657-ac13-459c-9197-e2a1557939b3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:09.677390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c7dc1657-ac13-459c-9197-e2a1557939b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:09.778597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dc1657-ac13-459c-9197-e2a1557939b3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:09.790597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:09.851314 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1111b7dc-86e3-4b19-9fa2-c140e00cd033 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:09.851917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1111b7dc-86e3-4b19-9fa2-c140e00cd033 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:09.852197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1111b7dc-86e3-4b19-9fa2-c140e00cd033 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:09.852471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1111b7dc-86e3-4b19-9fa2-c140e00cd033 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:09.854115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1111b7dc-86e3-4b19-9fa2-c140e00cd033 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:09.962212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1111b7dc-86e3-4b19-9fa2-c140e00cd033 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:09.963557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1111b7dc-86e3-4b19-9fa2-c140e00cd033 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:09.981638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 726/1446] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:12.235003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c47952-874f-4e62-ab69-ca3fc1a9169d None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:12.235449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c47952-874f-4e62-ab69-ca3fc1a9169d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:12.235558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c47952-874f-4e62-ab69-ca3fc1a9169d None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:12.236929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 721/1447] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:12.243587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc59e6-608a-4b11-a21a-a9e78b55b66d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:12.244045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc59e6-608a-4b11-a21a-a9e78b55b66d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:12.244154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bc59e6-608a-4b11-a21a-a9e78b55b66d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:12.291074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:12.309175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c3bc59e6-608a-4b11-a21a-a9e78b55b66d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:12.310397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c3bc59e6-608a-4b11-a21a-a9e78b55b66d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:12.403843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 727/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:12.683048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c47952-874f-4e62-ab69-ca3fc1a9169d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:12.684600 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-78c47952-874f-4e62-ab69-ca3fc1a9169d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:12.793896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c47952-874f-4e62-ab69-ca3fc1a9169d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:12.833324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:12.867902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c3fa1a-4864-4a3e-8897-a89c3ea63afc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:12.867902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c3fa1a-4864-4a3e-8897-a89c3ea63afc service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:12.868234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c3fa1a-4864-4a3e-8897-a89c3ea63afc service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:12.868234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c3fa1a-4864-4a3e-8897-a89c3ea63afc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:12.869820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-24c3fa1a-4864-4a3e-8897-a89c3ea63afc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:12.975708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c3fa1a-4864-4a3e-8897-a89c3ea63afc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:12.977037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c3fa1a-4864-4a3e-8897-a89c3ea63afc service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:12.995423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 722/1449] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:14.998550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f29d16b-f10b-4f56-90aa-3b0c762ec0f0 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:14.999136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f29d16b-f10b-4f56-90aa-3b0c762ec0f0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:14.999136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f29d16b-f10b-4f56-90aa-3b0c762ec0f0 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:15.000424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 728/1450] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:15.005713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213277a-ff25-49f2-b374-ff954289731b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:15.006139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213277a-ff25-49f2-b374-ff954289731b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:15.006808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c213277a-ff25-49f2-b374-ff954289731b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:15.071343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c213277a-ff25-49f2-b374-ff954289731b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:15.072531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c213277a-ff25-49f2-b374-ff954289731b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:15.093142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:15.168160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 723/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:15.448639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f29d16b-f10b-4f56-90aa-3b0c762ec0f0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:15.450271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9f29d16b-f10b-4f56-90aa-3b0c762ec0f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:15.554038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f29d16b-f10b-4f56-90aa-3b0c762ec0f0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:15.573235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:15.627392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0360bf-e7d4-4174-9fcf-d16543e9471c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:15.628003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0360bf-e7d4-4174-9fcf-d16543e9471c service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:15.628282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0360bf-e7d4-4174-9fcf-d16543e9471c service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:15.628555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0360bf-e7d4-4174-9fcf-d16543e9471c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:15.630169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1e0360bf-e7d4-4174-9fcf-d16543e9471c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:15.739361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0360bf-e7d4-4174-9fcf-d16543e9471c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:15.740766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0360bf-e7d4-4174-9fcf-d16543e9471c service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:15.758665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 729/1452] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:17.710656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8abce0-ac62-4d21-94ee-7416107d2804 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:17.711213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8abce0-ac62-4d21-94ee-7416107d2804 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:17.711213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8abce0-ac62-4d21-94ee-7416107d2804 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:17.712553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 724/1453] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:17.718035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534e98d1-a4e7-4ea2-9549-e289192d120a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:17.718444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534e98d1-a4e7-4ea2-9549-e289192d120a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:17.718586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534e98d1-a4e7-4ea2-9549-e289192d120a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:17.781274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:17.784337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-534e98d1-a4e7-4ea2-9549-e289192d120a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:17.785567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-534e98d1-a4e7-4ea2-9549-e289192d120a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:17.887123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 730/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:18.178827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8abce0-ac62-4d21-94ee-7416107d2804 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:18.180381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-af8abce0-ac62-4d21-94ee-7416107d2804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:18.285009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8abce0-ac62-4d21-94ee-7416107d2804 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:18.333317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:18.358346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c0c34a-0a53-4a23-91c9-b1857c688a3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:18.358952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c0c34a-0a53-4a23-91c9-b1857c688a3c service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:18.359225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c0c34a-0a53-4a23-91c9-b1857c688a3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:18.359492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c0c34a-0a53-4a23-91c9-b1857c688a3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:18.361063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b3c0c34a-0a53-4a23-91c9-b1857c688a3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:18.471653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c0c34a-0a53-4a23-91c9-b1857c688a3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:18.473135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c0c34a-0a53-4a23-91c9-b1857c688a3c service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:18.491747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 725/1455] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:20.437074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cd68ec-4fb8-4eaf-8899-58f4281977a8 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:20.437634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cd68ec-4fb8-4eaf-8899-58f4281977a8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:20.437728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cd68ec-4fb8-4eaf-8899-58f4281977a8 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:20.439029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 731/1456] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 05:07:20.444864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd28bb-5b05-4aa5-9a3a-30d00bdb8fcf None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:20.445299 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd28bb-5b05-4aa5-9a3a-30d00bdb8fcf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:20.445405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd28bb-5b05-4aa5-9a3a-30d00bdb8fcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:20.511393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-83fd28bb-5b05-4aa5-9a3a-30d00bdb8fcf None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:20.512529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-83fd28bb-5b05-4aa5-9a3a-30d00bdb8fcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:20.541326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:20.608030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 726/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:20.891885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cd68ec-4fb8-4eaf-8899-58f4281977a8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:20.893447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-57cd68ec-4fb8-4eaf-8899-58f4281977a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:20.998106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cd68ec-4fb8-4eaf-8899-58f4281977a8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:21.026481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:21.071568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78793f-9a9d-420c-8124-717a81060ed5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:21.072105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78793f-9a9d-420c-8124-717a81060ed5 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:21.072382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78793f-9a9d-420c-8124-717a81060ed5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:21.072657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78793f-9a9d-420c-8124-717a81060ed5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:21.074268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8a78793f-9a9d-420c-8124-717a81060ed5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:21.182131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a78793f-9a9d-420c-8124-717a81060ed5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:21.183556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a78793f-9a9d-420c-8124-717a81060ed5 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:21.201917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 732/1458] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:23.145915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4436094a-7434-4211-8857-31798df42acb None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:23.146490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4436094a-7434-4211-8857-31798df42acb None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:23.146490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4436094a-7434-4211-8857-31798df42acb None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:23.147811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 727/1459] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:23.153330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4ad7a-40e2-4037-9863-18cd0bf2de94 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:23.153766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4ad7a-40e2-4037-9863-18cd0bf2de94 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:23.153911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4ad7a-40e2-4037-9863-18cd0bf2de94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:23.218908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3ab4ad7a-40e2-4037-9863-18cd0bf2de94 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:23.220112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3ab4ad7a-40e2-4037-9863-18cd0bf2de94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:23.229099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:23.316036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 733/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:23.596030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4436094a-7434-4211-8857-31798df42acb None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:23.597580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4436094a-7434-4211-8857-31798df42acb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:23.702056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4436094a-7434-4211-8857-31798df42acb None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:23.757089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:23.776570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5778e7e-86eb-48e1-83f8-8eceb28b764f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:23.777200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5778e7e-86eb-48e1-83f8-8eceb28b764f service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:23.777476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5778e7e-86eb-48e1-83f8-8eceb28b764f service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:23.777789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5778e7e-86eb-48e1-83f8-8eceb28b764f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:23.779372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a5778e7e-86eb-48e1-83f8-8eceb28b764f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:23.885813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5778e7e-86eb-48e1-83f8-8eceb28b764f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:23.887242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5778e7e-86eb-48e1-83f8-8eceb28b764f service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:23.905281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 728/1461] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:25.842949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca202eb-1bb2-46e5-aad7-f072b06a0e61 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:25.843525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca202eb-1bb2-46e5-aad7-f072b06a0e61 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:25.843525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca202eb-1bb2-46e5-aad7-f072b06a0e61 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:25.844896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 734/1462] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:25.850062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b1c6d-0837-40b4-8940-a96af6a2f869 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:25.850467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b1c6d-0837-40b4-8940-a96af6a2f869 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:25.850646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921b1c6d-0837-40b4-8940-a96af6a2f869 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:25.915983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-921b1c6d-0837-40b4-8940-a96af6a2f869 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:25.917190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-921b1c6d-0837-40b4-8940-a96af6a2f869 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:25.954849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:26.011985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 729/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:26.290109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca202eb-1bb2-46e5-aad7-f072b06a0e61 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:26.291663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4ca202eb-1bb2-46e5-aad7-f072b06a0e61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:26.397747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca202eb-1bb2-46e5-aad7-f072b06a0e61 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:26.436933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:26.475069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5900af12-e179-47be-8da2-0aec0d0b6da3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:26.475823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5900af12-e179-47be-8da2-0aec0d0b6da3 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:26.475948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5900af12-e179-47be-8da2-0aec0d0b6da3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:26.476168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5900af12-e179-47be-8da2-0aec0d0b6da3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:26.477851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5900af12-e179-47be-8da2-0aec0d0b6da3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:26.586659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5900af12-e179-47be-8da2-0aec0d0b6da3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:26.588066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5900af12-e179-47be-8da2-0aec0d0b6da3 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:26.606565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 735/1464] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:28.503986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e488182e-dc58-412f-8925-aa04d933830e None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:28.504498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e488182e-dc58-412f-8925-aa04d933830e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:28.504498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e488182e-dc58-412f-8925-aa04d933830e None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:28.505888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 730/1465] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:28.511051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898ae4f7-5ddb-49ba-9815-e901ec37f40a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:28.511466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898ae4f7-5ddb-49ba-9815-e901ec37f40a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:28.511572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898ae4f7-5ddb-49ba-9815-e901ec37f40a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:28.577222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-898ae4f7-5ddb-49ba-9815-e901ec37f40a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:28.578438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-898ae4f7-5ddb-49ba-9815-e901ec37f40a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:28.596781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:28.677142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 736/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:28.954777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e488182e-dc58-412f-8925-aa04d933830e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:28.956330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e488182e-dc58-412f-8925-aa04d933830e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:29.059015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e488182e-dc58-412f-8925-aa04d933830e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:29.124385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:29.130837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff17aeba-83a6-4ca9-a18f-facaf1c5f980 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:29.131443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff17aeba-83a6-4ca9-a18f-facaf1c5f980 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:29.131725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff17aeba-83a6-4ca9-a18f-facaf1c5f980 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:29.132117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff17aeba-83a6-4ca9-a18f-facaf1c5f980 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:29.133694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ff17aeba-83a6-4ca9-a18f-facaf1c5f980 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:29.246307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff17aeba-83a6-4ca9-a18f-facaf1c5f980 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:29.247744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff17aeba-83a6-4ca9-a18f-facaf1c5f980 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:29.265275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 731/1467] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:31.175947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad0c8fa-53a2-4360-8ad5-3bbd00cb7352 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:31.176504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad0c8fa-53a2-4360-8ad5-3bbd00cb7352 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:31.176504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad0c8fa-53a2-4360-8ad5-3bbd00cb7352 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:31.177924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 737/1468] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:07:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:31.183232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca61fe52-f1e6-4056-be21-2672a261e115 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:31.183669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca61fe52-f1e6-4056-be21-2672a261e115 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:31.183805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca61fe52-f1e6-4056-be21-2672a261e115 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:31.252158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ca61fe52-f1e6-4056-be21-2672a261e115 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:31.253349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ca61fe52-f1e6-4056-be21-2672a261e115 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:31.301622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:31.350440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 732/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:07:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:31.635685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad0c8fa-53a2-4360-8ad5-3bbd00cb7352 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:31.637244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8ad0c8fa-53a2-4360-8ad5-3bbd00cb7352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:31.740268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad0c8fa-53a2-4360-8ad5-3bbd00cb7352 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:31.786931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:31.812687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30af7eea-6c3b-4975-8e5b-c586af2a2750 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:31.813291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30af7eea-6c3b-4975-8e5b-c586af2a2750 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:31.813566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30af7eea-6c3b-4975-8e5b-c586af2a2750 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:31.813867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30af7eea-6c3b-4975-8e5b-c586af2a2750 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:31.815444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-30af7eea-6c3b-4975-8e5b-c586af2a2750 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:31.921238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30af7eea-6c3b-4975-8e5b-c586af2a2750 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:31.922603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30af7eea-6c3b-4975-8e5b-c586af2a2750 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:31.941917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 738/1470] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:41.366819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=22 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:41.939674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=66 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:51.336825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298f1fc7-1726-466e-b8d1-d7163c13e345 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:51.337332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298f1fc7-1726-466e-b8d1-d7163c13e345 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:51.337332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298f1fc7-1726-466e-b8d1-d7163c13e345 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:51.338657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 733/1471] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:07:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:51.344144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1f253-37e4-43ba-90e0-0075761dc8fd None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:51.344534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1f253-37e4-43ba-90e0-0075761dc8fd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:51.344717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1f253-37e4-43ba-90e0-0075761dc8fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:51.410610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d3c1f253-37e4-43ba-90e0-0075761dc8fd None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:51.411892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d3c1f253-37e4-43ba-90e0-0075761dc8fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:51.508656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 739/1472] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:07:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:07:51.803157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ca63a-e56f-41e6-8da5-900c3fee44f5 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:51.803439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ca63a-e56f-41e6-8da5-900c3fee44f5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:51.803708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2ca63a-e56f-41e6-8da5-900c3fee44f5 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:51.805018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 734/1473] 10.222.0.16 () {54 vars in 967 bytes} [Thu Aug 25 05:07:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:51.809936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395268f3-5b71-4baf-a7b9-4167853dff95 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:51.810229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395268f3-5b71-4baf-a7b9-4167853dff95 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:51.810509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395268f3-5b71-4baf-a7b9-4167853dff95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:51.886559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-395268f3-5b71-4baf-a7b9-4167853dff95 None None] MFA Rules not processed for user `a6cc8f34f9674a94ba23a7157e44a6f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:51.887746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-395268f3-5b71-4baf-a7b9-4167853dff95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:51.907586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:51.962511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 740/1474] 10.222.0.16 () {58 vars in 1064 bytes} [Thu Aug 25 05:07:51 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:51.970345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2ca63a-e56f-41e6-8da5-900c3fee44f5 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:51.971910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2b2ca63a-e56f-41e6-8da5-900c3fee44f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:52.073850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2ca63a-e56f-41e6-8da5-900c3fee44f5 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:52.146524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9b0864-6207-4eec-b253-4922bd762f2a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6cc8f34f9674a94ba23a7157e44a6f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:52.147122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b0864-6207-4eec-b253-4922bd762f2a service glance] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:52.147398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b0864-6207-4eec-b253-4922bd762f2a service glance] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:52.147667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b0864-6207-4eec-b253-4922bd762f2a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:52.149219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7b9b0864-6207-4eec-b253-4922bd762f2a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:52.201408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:52.255955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9b0864-6207-4eec-b253-4922bd762f2a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:52.257386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9b0864-6207-4eec-b253-4922bd762f2a service glance] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:52.275852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 735/1475] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:54.230801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1159beb-ef3f-4041-8568-4818cb8a2c6d None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:54.231262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1159beb-ef3f-4041-8568-4818cb8a2c6d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:54.231382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1159beb-ef3f-4041-8568-4818cb8a2c6d None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:54.232689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 741/1476] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:07:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:54.238005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04fc2f-6a50-4937-a0e2-4ac7a0634f93 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:54.238437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04fc2f-6a50-4937-a0e2-4ac7a0634f93 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:54.238549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04fc2f-6a50-4937-a0e2-4ac7a0634f93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:54.303822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-dc04fc2f-6a50-4937-a0e2-4ac7a0634f93 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:54.305010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-dc04fc2f-6a50-4937-a0e2-4ac7a0634f93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:54.399033 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 736/1477] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:07:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:54.679186 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1159beb-ef3f-4041-8568-4818cb8a2c6d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:54.680760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a1159beb-ef3f-4041-8568-4818cb8a2c6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:54.805867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1159beb-ef3f-4041-8568-4818cb8a2c6d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:54.853618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:54.896741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929a7ac7-0833-4647-bc33-4cc31ed4189c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:54.897634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a7ac7-0833-4647-bc33-4cc31ed4189c admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:54.898060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a7ac7-0833-4647-bc33-4cc31ed4189c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:54.898394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a7ac7-0833-4647-bc33-4cc31ed4189c admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:54.899031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929a7ac7-0833-4647-bc33-4cc31ed4189c admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:54.900728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-929a7ac7-0833-4647-bc33-4cc31ed4189c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:07:54.900728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Aug 25 05:07:54.900728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: Aug 25 05:07:54.901883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929a7ac7-0833-4647-bc33-4cc31ed4189c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:54.921359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 742/1478] 10.222.0.16 () {56 vars in 1108 bytes} [Thu Aug 25 05:07:54 2022] GET /identity/v3/projects => generated 2553 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:07:56.840709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4140a97e-95fe-4c6b-9a55-f564d802fc0e None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:56.841531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4140a97e-95fe-4c6b-9a55-f564d802fc0e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:56.841531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4140a97e-95fe-4c6b-9a55-f564d802fc0e None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:56.842915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 737/1479] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:07:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:56.848231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc04029-2747-404e-bd69-c4ada3c4444a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:56.848653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc04029-2747-404e-bd69-c4ada3c4444a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:56.848790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc04029-2747-404e-bd69-c4ada3c4444a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:56.915127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-abc04029-2747-404e-bd69-c4ada3c4444a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:56.916353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-abc04029-2747-404e-bd69-c4ada3c4444a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:57.011637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 743/1480] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:07:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:07:57.302385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4140a97e-95fe-4c6b-9a55-f564d802fc0e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:07:57.303920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4140a97e-95fe-4c6b-9a55-f564d802fc0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:57.314514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:57.449556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4140a97e-95fe-4c6b-9a55-f564d802fc0e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:07:57.528446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:57.561552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c34e6f-f762-4717-8280-6421b9e7b0cc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:07:57.562191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c34e6f-f762-4717-8280-6421b9e7b0cc service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:57.562471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c34e6f-f762-4717-8280-6421b9e7b0cc service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:57.562741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c34e6f-f762-4717-8280-6421b9e7b0cc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:57.564335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-24c34e6f-f762-4717-8280-6421b9e7b0cc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:57.652657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c34e6f-f762-4717-8280-6421b9e7b0cc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:07:57.654373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c34e6f-f762-4717-8280-6421b9e7b0cc service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:07:57.672502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 738/1481] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:07:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:07:59.738434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03d14e-89ae-4902-b5eb-3784bbc70aa3 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:59.739093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03d14e-89ae-4902-b5eb-3784bbc70aa3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:59.739093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd03d14e-89ae-4902-b5eb-3784bbc70aa3 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:59.740373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 744/1482] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:07:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:07:59.745540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3959072-bbe0-45f7-be4e-a8c62aaaf150 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:07:59.745959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3959072-bbe0-45f7-be4e-a8c62aaaf150 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:07:59.746118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3959072-bbe0-45f7-be4e-a8c62aaaf150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:07:59.816355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c3959072-bbe0-45f7-be4e-a8c62aaaf150 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:07:59.817558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c3959072-bbe0-45f7-be4e-a8c62aaaf150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:07:59.863934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:07:59.913198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 739/1483] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:07:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:08:00.199577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd03d14e-89ae-4902-b5eb-3784bbc70aa3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:08:00.201103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fd03d14e-89ae-4902-b5eb-3784bbc70aa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:00.225674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:08:00.305130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd03d14e-89ae-4902-b5eb-3784bbc70aa3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:08:00.377448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d34356-6015-471b-b65f-f0e7490e87e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:08:00.378087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d34356-6015-471b-b65f-f0e7490e87e2 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:00.378371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d34356-6015-471b-b65f-f0e7490e87e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:00.378648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d34356-6015-471b-b65f-f0e7490e87e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:00.380771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-57d34356-6015-471b-b65f-f0e7490e87e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:00.454584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:08:00.489015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d34356-6015-471b-b65f-f0e7490e87e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:08:00.491296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d34356-6015-471b-b65f-f0e7490e87e2 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:08:00.509939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 745/1484] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:08:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:08:02.453059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86080bd-5621-4474-a541-f37a02e8eed1 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:02.453636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86080bd-5621-4474-a541-f37a02e8eed1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:02.453636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86080bd-5621-4474-a541-f37a02e8eed1 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:02.454965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 740/1485] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:08:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:08:02.460704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14c574-ee5e-4977-8c57-89828c349c1a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:02.461125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14c574-ee5e-4977-8c57-89828c349c1a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:02.461254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14c574-ee5e-4977-8c57-89828c349c1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:02.525520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-df14c574-ee5e-4977-8c57-89828c349c1a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:08:02.526666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-df14c574-ee5e-4977-8c57-89828c349c1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:02.620450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 746/1486] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:08:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:08:02.906658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86080bd-5621-4474-a541-f37a02e8eed1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:08:02.908192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c86080bd-5621-4474-a541-f37a02e8eed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:03.010720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86080bd-5621-4474-a541-f37a02e8eed1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:08:03.081988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:08:03.083768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326c6534-4df7-4735-b237-9de2a34b30da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:08:03.084369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326c6534-4df7-4735-b237-9de2a34b30da service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:03.084644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326c6534-4df7-4735-b237-9de2a34b30da service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:03.084915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326c6534-4df7-4735-b237-9de2a34b30da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:03.086534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-326c6534-4df7-4735-b237-9de2a34b30da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:03.193672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326c6534-4df7-4735-b237-9de2a34b30da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:08:03.195046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326c6534-4df7-4735-b237-9de2a34b30da service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:08:03.212979 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 741/1487] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:08:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:08:05.255339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8be103-8477-454b-b841-5c01b4bd5579 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:05.255839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8be103-8477-454b-b841-5c01b4bd5579 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:05.255953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8be103-8477-454b-b841-5c01b4bd5579 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:05.257270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 747/1488] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:08:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:08:05.262354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3fab63-56bf-45cb-bac0-0732effdf68c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:05.262766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3fab63-56bf-45cb-bac0-0732effdf68c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:05.262879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3fab63-56bf-45cb-bac0-0732effdf68c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:05.328012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-bf3fab63-56bf-45cb-bac0-0732effdf68c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:08:05.329137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-bf3fab63-56bf-45cb-bac0-0732effdf68c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:05.383212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:08:05.426663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 742/1489] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:08:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:08:05.714190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8be103-8477-454b-b841-5c01b4bd5579 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:08:05.716010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-db8be103-8477-454b-b841-5c01b4bd5579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:05.752263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:08:05.818571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8be103-8477-454b-b841-5c01b4bd5579 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:08:05.890545 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dd2f2a-5e50-46bf-a39b-2e53b16acd91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:08:05.891147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd2f2a-5e50-46bf-a39b-2e53b16acd91 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:05.891427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd2f2a-5e50-46bf-a39b-2e53b16acd91 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:05.891698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dd2f2a-5e50-46bf-a39b-2e53b16acd91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:05.893277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c6dd2f2a-5e50-46bf-a39b-2e53b16acd91 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:05.975528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:08:06.003705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd2f2a-5e50-46bf-a39b-2e53b16acd91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:08:06.005224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dd2f2a-5e50-46bf-a39b-2e53b16acd91 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:08:06.024302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 748/1490] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:08:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:08:08.404123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13d940a-b790-41fd-bee1-ba9b0ed765cc None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:08.404677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13d940a-b790-41fd-bee1-ba9b0ed765cc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:08.404677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13d940a-b790-41fd-bee1-ba9b0ed765cc None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:08.406062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 743/1491] 10.222.0.16 () {54 vars in 927 bytes} [Thu Aug 25 05:08:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:08:08.411752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88767609-f434-46f3-af0e-ce4523863f7e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:08.412169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88767609-f434-46f3-af0e-ce4523863f7e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:08.412324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88767609-f434-46f3-af0e-ce4523863f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:08.477851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-88767609-f434-46f3-af0e-ce4523863f7e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:08:08.478968 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-88767609-f434-46f3-af0e-ce4523863f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:08.572308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 749/1492] 10.222.0.16 () {58 vars in 1024 bytes} [Thu Aug 25 05:08:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:08:08.860558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13d940a-b790-41fd-bee1-ba9b0ed765cc None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:08:08.862150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a13d940a-b790-41fd-bee1-ba9b0ed765cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:08.965450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13d940a-b790-41fd-bee1-ba9b0ed765cc None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:08:09.038595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ab0934-2eb4-4edf-a2d2-a3d8d592ac93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:08:09.039208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab0934-2eb4-4edf-a2d2-a3d8d592ac93 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:08:09.039488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab0934-2eb4-4edf-a2d2-a3d8d592ac93 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:08:09.039762 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ab0934-2eb4-4edf-a2d2-a3d8d592ac93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:08:09.041356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-45ab0934-2eb4-4edf-a2d2-a3d8d592ac93 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:08:09.050719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:08:09.150458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ab0934-2eb4-4edf-a2d2-a3d8d592ac93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:08:09.151899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ab0934-2eb4-4edf-a2d2-a3d8d592ac93 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:08:09.177739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 744/1493] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:08:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:08:18.570603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=82 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:08:19.174854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=50 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:56.794183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77580e14-ccac-4af9-9dd6-531ebc2f22fb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:56.794771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77580e14-ccac-4af9-9dd6-531ebc2f22fb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:56.794771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77580e14-ccac-4af9-9dd6-531ebc2f22fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:56.869734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-77580e14-ccac-4af9-9dd6-531ebc2f22fb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:09:56.870927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-77580e14-ccac-4af9-9dd6-531ebc2f22fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:56.968046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 750/1494] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:09:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:09:56.976222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f0983f-752b-4203-af59-a31539f90140 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:56.976704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f0983f-752b-4203-af59-a31539f90140 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:56.976812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f0983f-752b-4203-af59-a31539f90140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:57.052500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-00f0983f-752b-4203-af59-a31539f90140 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:09:57.053637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-00f0983f-752b-4203-af59-a31539f90140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:57.150447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 745/1495] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:09:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:09:57.155393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77580e14-ccac-4af9-9dd6-531ebc2f22fb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:57.163645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-77580e14-ccac-4af9-9dd6-531ebc2f22fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:57.168327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77580e14-ccac-4af9-9dd6-531ebc2f22fb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:57.230685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:57.264813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:57.265421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:57.265730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:57.266006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:57.266649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:09:57.266969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:09:57.268425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:09:57.268425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Aug 25 05:09:57.268425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: Aug 25 05:09:57.269445 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:09:57.280699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 751/1496] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:09:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:09:57.289536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f0983f-752b-4203-af59-a31539f90140 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:57.298500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f0983f-752b-4203-af59-a31539f90140 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:57.358850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:57.392271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:57.392905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:57.393853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:57.394130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:57.394754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:09:57.396530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WARNING py.warnings [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:09:57.396530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: warnings.warn(msg) Aug 25 05:09:57.396530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: Aug 25 05:09:57.397759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:09:57.532157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 746/1497] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:09:57 2022] POST /identity/v3/projects => generated 377 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:09:57.537347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:57.539114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:57.571770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f97fce-6d31-42a8-ac25-79002543f6b9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:57.663228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:57.663823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:57.664113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:57.664361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:57.664967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:09:57.665987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:09:57.665987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Aug 25 05:09:57.665987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: Aug 25 05:09:57.667176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:09:57.771678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 752/1498] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:09:57 2022] POST /identity/v3/users => generated 289 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:09:57.776078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:57.777570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:57.839312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d44626-cf35-4c38-a4a1-3d95cfcf945f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:57.920593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:57.931043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:57.931658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:57.931938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:57.932207 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:57.932843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:09:57.933889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WARNING py.warnings [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:09:57.933889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: warnings.warn(msg) Aug 25 05:09:57.933889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: Aug 25 05:09:57.934770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:09:57.947800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 747/1499] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:09:57 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:09:57.951827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:57.953390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:57.965482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea4dcc9-180f-4fbf-b800-878d9fa523ff admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:57.991063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:58.060851 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a789b4-8362-4f5e-9313-231bfe93dd4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:58.061464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a789b4-8362-4f5e-9313-231bfe93dd4d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:58.061776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a789b4-8362-4f5e-9313-231bfe93dd4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:58.062051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a789b4-8362-4f5e-9313-231bfe93dd4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:58.062691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a789b4-8362-4f5e-9313-231bfe93dd4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:09:58.063695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-00a789b4-8362-4f5e-9313-231bfe93dd4d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:09:58.063695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Aug 25 05:09:58.063695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: Aug 25 05:09:58.064583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a789b4-8362-4f5e-9313-231bfe93dd4d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:09:58.077562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 753/1500] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:09:57 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:09:58.082577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:58.084101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:58.117414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e262e50-9cd1-4d13-a677-3d451826572e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:58.209895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93684584-dfad-4d23-a91e-fab661fffcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:58.210478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93684584-dfad-4d23-a91e-fab661fffcd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:58.210753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93684584-dfad-4d23-a91e-fab661fffcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:58.211021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93684584-dfad-4d23-a91e-fab661fffcd3 admin admin] PATH_INFO: `/v3/projects/70b12767a9bd4779b6b02d43cbe23640/users/21273740f5a14e628847b487478869e1/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:58.233300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93684584-dfad-4d23-a91e-fab661fffcd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=70b12767a9bd4779b6b02d43cbe23640, user_id=21273740f5a14e628847b487478869e1, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:09:58.234378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WARNING py.warnings [None req-93684584-dfad-4d23-a91e-fab661fffcd3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:09:58.234378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: warnings.warn(msg) Aug 25 05:09:58.234378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: Aug 25 05:09:58.236327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93684584-dfad-4d23-a91e-fab661fffcd3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:09:58.355060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 748/1501] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:09:58 2022] PUT /identity/v3/projects/70b12767a9bd4779b6b02d43cbe23640/users/21273740f5a14e628847b487478869e1/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:09:58.367720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb176e5-980f-41a3-a4be-70d7bdca39f3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:58.368005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb176e5-980f-41a3-a4be-70d7bdca39f3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:58.368278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb176e5-980f-41a3-a4be-70d7bdca39f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:58.426724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-abb176e5-980f-41a3-a4be-70d7bdca39f3 None None] MFA Rules not processed for user `21273740f5a14e628847b487478869e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:09:58.427923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-abb176e5-980f-41a3-a4be-70d7bdca39f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:58.505198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 754/1502] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:09:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:09:58.518020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fe0a6-38f7-4102-833a-62c9653f0bf4 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:58.518308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fe0a6-38f7-4102-833a-62c9653f0bf4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:58.518584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fe0a6-38f7-4102-833a-62c9653f0bf4 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:58.519908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 749/1503] 10.222.0.16 () {54 vars in 964 bytes} [Thu Aug 25 05:09:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:09:58.524512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217d71db-f160-4deb-b17b-8ddafa9c458d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:58.524802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217d71db-f160-4deb-b17b-8ddafa9c458d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:58.525080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217d71db-f160-4deb-b17b-8ddafa9c458d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:58.557800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:58.604482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-217d71db-f160-4deb-b17b-8ddafa9c458d None None] MFA Rules not processed for user `ec312d29dfa846cbace53e8444b8a527`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:09:58.605709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-217d71db-f160-4deb-b17b-8ddafa9c458d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:58.681270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 755/1504] 10.222.0.16 () {58 vars in 1061 bytes} [Thu Aug 25 05:09:58 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:09:58.689197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42fe0a6-38f7-4102-833a-62c9653f0bf4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:58.690859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a42fe0a6-38f7-4102-833a-62c9653f0bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:58.741667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:58.794458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42fe0a6-38f7-4102-833a-62c9653f0bf4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:58.867370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:58.867973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:58.868247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:58.868518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:58.870129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:58.966007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:09:58.967428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:09:58.970838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:58.985750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 750/1505] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:09:58 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:09:59.030276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0acd55-b7ec-4362-9a9f-51593cf2fe6f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:59.030721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0acd55-b7ec-4362-9a9f-51593cf2fe6f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:59.030920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0acd55-b7ec-4362-9a9f-51593cf2fe6f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:59.032126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 756/1506] 10.222.0.16 () {52 vars in 830 bytes} [Thu Aug 25 05:09:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:09:59.036354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:59.037558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:59.135096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d507d34-e1d4-471a-a7dc-15efb15bcaf9 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:59.229505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:59.230160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:59.230433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:59.230731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] PATH_INFO: `/v3/users/21273740f5a14e628847b487478869e1` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:59.243508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=21273740f5a14e628847b487478869e1)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:09:59.244588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WARNING py.warnings [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:09:59.244588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: warnings.warn(msg) Aug 25 05:09:59.244588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: Aug 25 05:09:59.246042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:09:59.256476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:59.490540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21273740f5a14e628847b487478869e1'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:09:59.513431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21273740f5a14e628847b487478869e1'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:09:59.513912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21273740f5a14e628847b487478869e1'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:09:59.591008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21273740f5a14e628847b487478869e1'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:09:59.605337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21273740f5a14e628847b487478869e1'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:09:59.611016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 751/1507] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:09:59 2022] DELETE /identity/v3/users/21273740f5a14e628847b487478869e1 => generated 0 bytes in 575 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:09:59.616506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef2e7cb-2ab6-43df-a14d-3abdc58a5f56 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:59.616790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef2e7cb-2ab6-43df-a14d-3abdc58a5f56 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:59.617063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef2e7cb-2ab6-43df-a14d-3abdc58a5f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:59.678457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bef2e7cb-2ab6-43df-a14d-3abdc58a5f56 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:09:59.679682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bef2e7cb-2ab6-43df-a14d-3abdc58a5f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:59.759877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:59.773556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 757/1508] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:09:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:09:59.783339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:09:59.784563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:09:59.885822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512cf888-2f46-41bc-997e-c5460056b0cb admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:09:59.940581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:09:59.959011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d60996-111b-424c-a283-e3c60e0cf35c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:09:59.959617 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d60996-111b-424c-a283-e3c60e0cf35c service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:09:59.959894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d60996-111b-424c-a283-e3c60e0cf35c service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:09:59.960163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d60996-111b-424c-a283-e3c60e0cf35c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:09:59.961765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-40d60996-111b-424c-a283-e3c60e0cf35c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:10:00.066811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d60996-111b-424c-a283-e3c60e0cf35c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:10:00.068165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d60996-111b-424c-a283-e3c60e0cf35c service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:10:00.086749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 752/1509] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:09:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:10:00.638260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef2e7cb-2ab6-43df-a14d-3abdc58a5f56 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:10:00.639709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bef2e7cb-2ab6-43df-a14d-3abdc58a5f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:10:00.651878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef2e7cb-2ab6-43df-a14d-3abdc58a5f56 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:10:00.656482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:10:00.657139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:10:00.657422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:10:00.657722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] PATH_INFO: `/v3/projects/70b12767a9bd4779b6b02d43cbe23640` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:10:00.670865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] RBAC: Authorizing `identity:delete_project(project_id=70b12767a9bd4779b6b02d43cbe23640)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:10:00.671893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:10:00.671893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Aug 25 05:10:00.671893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: Aug 25 05:10:00.673094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:10:00.801301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '70b12767a9bd4779b6b02d43cbe23640', 'request_id': 'req-493e312e-488d-4b12-8e63-939f912dc663'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:10:00.807783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '70b12767a9bd4779b6b02d43cbe23640', 'request_id': 'req-493e312e-488d-4b12-8e63-939f912dc663'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:10:00.888434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '70b12767a9bd4779b6b02d43cbe23640', 'request_id': 'req-493e312e-488d-4b12-8e63-939f912dc663'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:10:00.907533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] The token cache is being invalidate because project 70b12767a9bd4779b6b02d43cbe23640 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:10:00.907819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:10:00.908456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-493e312e-488d-4b12-8e63-939f912dc663 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:10:00.910071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 758/1510] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:10:00 2022] DELETE /identity/v3/projects/70b12767a9bd4779b6b02d43cbe23640 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:10:10.084537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=62 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:10:10.904498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=36,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:11:26.990391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e8320-5aa0-4015-bbe6-8717c04006f1 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:26.990987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e8320-5aa0-4015-bbe6-8717c04006f1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:26.990987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e8320-5aa0-4015-bbe6-8717c04006f1 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:26.992290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 753/1511] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:11:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:11:26.996980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2936a57-2c45-4731-bb6f-e9f8682d3486 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:26.997394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2936a57-2c45-4731-bb6f-e9f8682d3486 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:26.997539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2936a57-2c45-4731-bb6f-e9f8682d3486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:27.048315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a2936a57-2c45-4731-bb6f-e9f8682d3486 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:27.049555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a2936a57-2c45-4731-bb6f-e9f8682d3486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:27.107002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 759/1512] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:11:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:27.264816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05e8320-5aa0-4015-bbe6-8717c04006f1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:27.266959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c05e8320-5aa0-4015-bbe6-8717c04006f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:27.386912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05e8320-5aa0-4015-bbe6-8717c04006f1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:27.392448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b391aa49-3a1e-41ce-9594-0568b2595f7f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:27.393057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391aa49-3a1e-41ce-9594-0568b2595f7f service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:27.393340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391aa49-3a1e-41ce-9594-0568b2595f7f service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:27.393704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391aa49-3a1e-41ce-9594-0568b2595f7f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:27.401875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b391aa49-3a1e-41ce-9594-0568b2595f7f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:27.403501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b391aa49-3a1e-41ce-9594-0568b2595f7f service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:27.412788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 754/1513] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:11:28.237904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef74bd0-4faf-4847-91a1-2d1472d9c1fb None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:28.238374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef74bd0-4faf-4847-91a1-2d1472d9c1fb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:28.238525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef74bd0-4faf-4847-91a1-2d1472d9c1fb None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:28.239866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 760/1514] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:11:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:11:28.244697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e7144-0fb9-402d-b100-814af15b3bed None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:28.245090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e7144-0fb9-402d-b100-814af15b3bed None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:28.245273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e7144-0fb9-402d-b100-814af15b3bed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:28.264319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f49e7144-0fb9-402d-b100-814af15b3bed None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:28.265512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f49e7144-0fb9-402d-b100-814af15b3bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:28.278458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 755/1515] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:11:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:28.430537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef74bd0-4faf-4847-91a1-2d1472d9c1fb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:28.445147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef74bd0-4faf-4847-91a1-2d1472d9c1fb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:28.450612 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b04388-c704-43e6-aa86-ed976a1827fa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:28.451201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b04388-c704-43e6-aa86-ed976a1827fa service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:28.451481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b04388-c704-43e6-aa86-ed976a1827fa service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:28.451756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b04388-c704-43e6-aa86-ed976a1827fa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:28.460321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b04388-c704-43e6-aa86-ed976a1827fa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:28.461928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b04388-c704-43e6-aa86-ed976a1827fa service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:28.471401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 761/1516] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:11:29.046405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232afd4b-8ec6-48e3-b498-7434181f8c51 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:29.046690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232afd4b-8ec6-48e3-b498-7434181f8c51 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:29.046967 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232afd4b-8ec6-48e3-b498-7434181f8c51 None None] PATH_INFO: `/v3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:29.048238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 756/1517] 10.222.0.47 () {54 vars in 925 bytes} [Thu Aug 25 05:11:29 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:11:29.056915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a119a-f6f9-4346-98f7-51cc4edf777a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:29.057206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a119a-f6f9-4346-98f7-51cc4edf777a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:29.057480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a119a-f6f9-4346-98f7-51cc4edf777a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:29.116208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a99a119a-f6f9-4346-98f7-51cc4edf777a None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:29.117397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a99a119a-f6f9-4346-98f7-51cc4edf777a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:29.187099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 762/1518] 10.222.0.47 () {58 vars in 1015 bytes} [Thu Aug 25 05:11:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:38.259616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=56 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:11:38.453811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b867f-a1e6-4e1b-8458-caf7d3bcd17f None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:38.454012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b867f-a1e6-4e1b-8458-caf7d3bcd17f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:38.454270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b867f-a1e6-4e1b-8458-caf7d3bcd17f None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:38.455637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 757/1519] 10.222.0.16 () {54 vars in 961 bytes} [Thu Aug 25 05:11:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:11:38.461299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684ad4c-9653-4f32-a5d8-86aa9cea95c5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:38.461781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684ad4c-9653-4f32-a5d8-86aa9cea95c5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:38.461893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e684ad4c-9653-4f32-a5d8-86aa9cea95c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:38.480932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e684ad4c-9653-4f32-a5d8-86aa9cea95c5 None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:38.482141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e684ad4c-9653-4f32-a5d8-86aa9cea95c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:38.491289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 763/1520] 10.222.0.16 () {58 vars in 1058 bytes} [Thu Aug 25 05:11:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:38.498592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5b867f-a1e6-4e1b-8458-caf7d3bcd17f None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:38.512431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5b867f-a1e6-4e1b-8458-caf7d3bcd17f None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:38.517318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7980e4-8936-4ecc-9ad7-9b30a69f9472 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:38.517942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7980e4-8936-4ecc-9ad7-9b30a69f9472 service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:38.518222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7980e4-8936-4ecc-9ad7-9b30a69f9472 service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:38.518494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7980e4-8936-4ecc-9ad7-9b30a69f9472 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:38.526547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7980e4-8936-4ecc-9ad7-9b30a69f9472 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:38.528116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7980e4-8936-4ecc-9ad7-9b30a69f9472 service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:38.537478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 758/1521] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:11:39.695231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918369f-13b5-4001-bf3c-e8cba65ae1eb None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:39.695802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918369f-13b5-4001-bf3c-e8cba65ae1eb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:39.695802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918369f-13b5-4001-bf3c-e8cba65ae1eb None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:39.697123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 764/1522] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:11:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:11:39.702014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed60aed-3226-4cce-8741-ede02c8e2726 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:39.702416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed60aed-3226-4cce-8741-ede02c8e2726 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:39.702568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed60aed-3226-4cce-8741-ede02c8e2726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:39.722278 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6ed60aed-3226-4cce-8741-ede02c8e2726 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:39.723474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6ed60aed-3226-4cce-8741-ede02c8e2726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:39.731177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 759/1523] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:11:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:39.882994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e918369f-13b5-4001-bf3c-e8cba65ae1eb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:39.897913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e918369f-13b5-4001-bf3c-e8cba65ae1eb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:39.903345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01da1b7-c1a5-4047-9e5b-8616aa621375 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:39.903977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01da1b7-c1a5-4047-9e5b-8616aa621375 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:39.904257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01da1b7-c1a5-4047-9e5b-8616aa621375 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:39.904531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01da1b7-c1a5-4047-9e5b-8616aa621375 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:39.912526 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01da1b7-c1a5-4047-9e5b-8616aa621375 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:39.914130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01da1b7-c1a5-4047-9e5b-8616aa621375 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:39.917962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:11:39.923808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 765/1524] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:11:40.742978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba9871f-aaf6-44e8-a195-b43e14a8c04c None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:40.743549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba9871f-aaf6-44e8-a195-b43e14a8c04c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:40.743549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba9871f-aaf6-44e8-a195-b43e14a8c04c None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:40.744874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 760/1525] 127.0.0.1 () {54 vars in 901 bytes} [Thu Aug 25 05:11:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:11:40.749571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf975925-5519-48f8-ac0d-22b7fa037a5a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:40.750026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf975925-5519-48f8-ac0d-22b7fa037a5a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:40.750143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf975925-5519-48f8-ac0d-22b7fa037a5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:40.768982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cf975925-5519-48f8-ac0d-22b7fa037a5a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:40.770684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cf975925-5519-48f8-ac0d-22b7fa037a5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:40.778298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 766/1526] 127.0.0.1 () {58 vars in 998 bytes} [Thu Aug 25 05:11:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:40.928578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba9871f-aaf6-44e8-a195-b43e14a8c04c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:40.944673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba9871f-aaf6-44e8-a195-b43e14a8c04c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:40.950419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a3a1d7-3821-475c-90a0-c3005dde31a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:40.951005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a3a1d7-3821-475c-90a0-c3005dde31a8 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:40.951284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a3a1d7-3821-475c-90a0-c3005dde31a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:40.951559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a3a1d7-3821-475c-90a0-c3005dde31a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:40.959961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a3a1d7-3821-475c-90a0-c3005dde31a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:40.961528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a3a1d7-3821-475c-90a0-c3005dde31a8 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:40.970959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 761/1527] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:11:44.778715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb1229-5573-4a84-a14c-e135aae2f566 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:44.779247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb1229-5573-4a84-a14c-e135aae2f566 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:44.779247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeeb1229-5573-4a84-a14c-e135aae2f566 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:44.780626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 767/1528] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:11:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:11:44.786082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9f587-691f-4759-b417-7427889c9529 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:44.786539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9f587-691f-4759-b417-7427889c9529 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:44.786646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9f587-691f-4759-b417-7427889c9529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:44.805794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c8b9f587-691f-4759-b417-7427889c9529 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:44.806970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c8b9f587-691f-4759-b417-7427889c9529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:44.814675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 762/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:11:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:45.107530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeb1229-5573-4a84-a14c-e135aae2f566 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:45.109247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-aeeb1229-5573-4a84-a14c-e135aae2f566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:45.187676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeeb1229-5573-4a84-a14c-e135aae2f566 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:45.191323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ee43ff-6598-4d0b-bd6c-529d223b4527 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:45.191927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee43ff-6598-4d0b-bd6c-529d223b4527 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:45.192210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee43ff-6598-4d0b-bd6c-529d223b4527 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:45.192481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee43ff-6598-4d0b-bd6c-529d223b4527 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:45.201181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee43ff-6598-4d0b-bd6c-529d223b4527 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:45.202664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ee43ff-6598-4d0b-bd6c-529d223b4527 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:45.212350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 768/1530] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:11:47.120865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bb6c83-7dc3-408e-8451-6f9ccca08533 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:47.121447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bb6c83-7dc3-408e-8451-6f9ccca08533 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:47.121447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bb6c83-7dc3-408e-8451-6f9ccca08533 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:47.122831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 763/1531] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:11:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:11:47.128712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1833e754-27c9-4abe-b5cd-2e2af1e0eeff None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:47.129126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1833e754-27c9-4abe-b5cd-2e2af1e0eeff None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:47.129304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1833e754-27c9-4abe-b5cd-2e2af1e0eeff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:47.149075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1833e754-27c9-4abe-b5cd-2e2af1e0eeff None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:47.150308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1833e754-27c9-4abe-b5cd-2e2af1e0eeff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:47.158201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 769/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:11:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:47.443371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bb6c83-7dc3-408e-8451-6f9ccca08533 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:47.454138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bb6c83-7dc3-408e-8451-6f9ccca08533 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:47.458282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a1a1f9-43be-46ed-821a-0d5ad1881ae8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:47.458873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a1a1f9-43be-46ed-821a-0d5ad1881ae8 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:47.459152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a1a1f9-43be-46ed-821a-0d5ad1881ae8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:47.459425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a1a1f9-43be-46ed-821a-0d5ad1881ae8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:47.468399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a1a1f9-43be-46ed-821a-0d5ad1881ae8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:47.469847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a1a1f9-43be-46ed-821a-0d5ad1881ae8 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:47.479307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 764/1533] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:11:50.791914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b9091-d28e-4cff-8a20-e484ea17f925 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:50.792476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b9091-d28e-4cff-8a20-e484ea17f925 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:50.792476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b9091-d28e-4cff-8a20-e484ea17f925 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:50.793907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 770/1534] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:11:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 05:11:50.799230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97c51f-e936-4917-8c29-4ad21d86a01e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:50.799654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97c51f-e936-4917-8c29-4ad21d86a01e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:50.799775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a97c51f-e936-4917-8c29-4ad21d86a01e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:50.818393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1a97c51f-e936-4917-8c29-4ad21d86a01e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:50.819567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1a97c51f-e936-4917-8c29-4ad21d86a01e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:50.827199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 765/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:11:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:51.380765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b9091-d28e-4cff-8a20-e484ea17f925 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:51.382522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-318b9091-d28e-4cff-8a20-e484ea17f925 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:51.463306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b9091-d28e-4cff-8a20-e484ea17f925 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:51.467090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c256a5d0-a050-4790-a914-8d64c98a04b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:51.467696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c256a5d0-a050-4790-a914-8d64c98a04b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:51.467974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c256a5d0-a050-4790-a914-8d64c98a04b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:51.468246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c256a5d0-a050-4790-a914-8d64c98a04b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:51.477052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c256a5d0-a050-4790-a914-8d64c98a04b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:51.478508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c256a5d0-a050-4790-a914-8d64c98a04b0 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:51.488234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 771/1536] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:11:53.400592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294b6a6-acc9-4dd8-b108-e8d633947b25 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:53.401144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294b6a6-acc9-4dd8-b108-e8d633947b25 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:53.401144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9294b6a6-acc9-4dd8-b108-e8d633947b25 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:53.402507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 766/1537] 127.0.0.1 () {54 vars in 917 bytes} [Thu Aug 25 05:11:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:11:53.407740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0d8fd-c07d-44e8-8790-5398761699f1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:53.408158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0d8fd-c07d-44e8-8790-5398761699f1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:53.408260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c0d8fd-c07d-44e8-8790-5398761699f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:53.427906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-91c0d8fd-c07d-44e8-8790-5398761699f1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:11:53.429161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-91c0d8fd-c07d-44e8-8790-5398761699f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:53.436964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 772/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Aug 25 05:11:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:11:53.983418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9294b6a6-acc9-4dd8-b108-e8d633947b25 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:11:53.985076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9294b6a6-acc9-4dd8-b108-e8d633947b25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:11:53.998429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9294b6a6-acc9-4dd8-b108-e8d633947b25 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:11:54.002081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cc69c5-cc38-4f75-89b0-d020d180d316 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:11:54.002674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc69c5-cc38-4f75-89b0-d020d180d316 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:11:54.002993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc69c5-cc38-4f75-89b0-d020d180d316 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:11:54.003264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cc69c5-cc38-4f75-89b0-d020d180d316 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:11:54.011986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cc69c5-cc38-4f75-89b0-d020d180d316 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:11:54.013399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cc69c5-cc38-4f75-89b0-d020d180d316 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:11:54.022528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 767/1539] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:11:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:12:03.422475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=80 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:12:04.018791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=33 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:17.445645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6802b34-a629-4977-9de7-131175a6e2d0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.446863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6802b34-a629-4977-9de7-131175a6e2d0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.447362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6802b34-a629-4977-9de7-131175a6e2d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.483739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a6802b34-a629-4977-9de7-131175a6e2d0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:17.486391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a6802b34-a629-4977-9de7-131175a6e2d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:17.499436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 773/1540] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:13:17.508271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eea1718-6515-4fd0-8832-13751d025b1e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.509451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eea1718-6515-4fd0-8832-13751d025b1e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.509981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eea1718-6515-4fd0-8832-13751d025b1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.535247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4eea1718-6515-4fd0-8832-13751d025b1e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:17.537179 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4eea1718-6515-4fd0-8832-13751d025b1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:17.553408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 768/1541] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:17.558148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6802b34-a629-4977-9de7-131175a6e2d0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:17.569499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6802b34-a629-4977-9de7-131175a6e2d0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:17.574871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603ffed0-2104-4e8f-a6be-01e36616c55e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:17.575824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603ffed0-2104-4e8f-a6be-01e36616c55e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.576328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603ffed0-2104-4e8f-a6be-01e36616c55e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.576811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603ffed0-2104-4e8f-a6be-01e36616c55e admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.577711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603ffed0-2104-4e8f-a6be-01e36616c55e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:17.579040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603ffed0-2104-4e8f-a6be-01e36616c55e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:17.580852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603ffed0-2104-4e8f-a6be-01e36616c55e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:17.598424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 774/1542] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:17.606983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eea1718-6515-4fd0-8832-13751d025b1e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:17.617366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eea1718-6515-4fd0-8832-13751d025b1e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:17.624570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63da9fe9-a3af-4114-9ccb-5d64b68a7c64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:17.625895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63da9fe9-a3af-4114-9ccb-5d64b68a7c64 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.626419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63da9fe9-a3af-4114-9ccb-5d64b68a7c64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.626933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63da9fe9-a3af-4114-9ccb-5d64b68a7c64 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.627974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63da9fe9-a3af-4114-9ccb-5d64b68a7c64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:17.630486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63da9fe9-a3af-4114-9ccb-5d64b68a7c64 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:17.644648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0370238c-5296-4f8c-887e-db84f4fd6c25 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.645323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0370238c-5296-4f8c-887e-db84f4fd6c25 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.645843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0370238c-5296-4f8c-887e-db84f4fd6c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.694852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0370238c-5296-4f8c-887e-db84f4fd6c25 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:17.696506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0370238c-5296-4f8c-887e-db84f4fd6c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:17.713784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 775/1543] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:17.725430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eed1e4-39ae-4067-9343-fc61e926dd1e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.726202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eed1e4-39ae-4067-9343-fc61e926dd1e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.726689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eed1e4-39ae-4067-9343-fc61e926dd1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.751725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d6eed1e4-39ae-4067-9343-fc61e926dd1e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:17.753439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d6eed1e4-39ae-4067-9343-fc61e926dd1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:17.768689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 776/1544] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:13:17.773808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eed1e4-39ae-4067-9343-fc61e926dd1e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:17.784654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eed1e4-39ae-4067-9343-fc61e926dd1e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:17.791572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e7846c-f2f2-47d7-9f29-a9cebf0f0451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:17.792601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e7846c-f2f2-47d7-9f29-a9cebf0f0451 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.793269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e7846c-f2f2-47d7-9f29-a9cebf0f0451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.793786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e7846c-f2f2-47d7-9f29-a9cebf0f0451 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.794666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e7846c-f2f2-47d7-9f29-a9cebf0f0451 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:17.795183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e7846c-f2f2-47d7-9f29-a9cebf0f0451 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:17.797210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e7846c-f2f2-47d7-9f29-a9cebf0f0451 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:17.819789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 769/1545] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:17 2022] POST /identity/v3/projects => generated 385 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:17.824227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63da9fe9-a3af-4114-9ccb-5d64b68a7c64 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:17.826701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 777/1546] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:13:17.828719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0acc5-3904-449c-8201-505276488c87 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.829017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0acc5-3904-449c-8201-505276488c87 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.829294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0acc5-3904-449c-8201-505276488c87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.867267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-50c0acc5-3904-449c-8201-505276488c87 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:17.868965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-50c0acc5-3904-449c-8201-505276488c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:17.960554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63da9fe9-a3af-4114-9ccb-5d64b68a7c64 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:17.967917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 778/1547] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:17.969011 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c0acc5-3904-449c-8201-505276488c87 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:17.974315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b52c05-daff-411c-8612-151912e57ce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:17.975295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b52c05-daff-411c-8612-151912e57ce5 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.975794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b52c05-daff-411c-8612-151912e57ce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.976404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b52c05-daff-411c-8612-151912e57ce5 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.977323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b52c05-daff-411c-8612-151912e57ce5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:17.978557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WARNING py.warnings [None req-b9b52c05-daff-411c-8612-151912e57ce5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:13:17.978557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: warnings.warn(msg) Aug 25 05:13:17.978557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: Aug 25 05:13:17.981511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b52c05-daff-411c-8612-151912e57ce5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:17.981640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c0acc5-3904-449c-8201-505276488c87 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:17.986398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b31dfee-8c28-41e2-8ce5-0d4baa1adebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:17.987246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31dfee-8c28-41e2-8ce5-0d4baa1adebb admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:17.987715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31dfee-8c28-41e2-8ce5-0d4baa1adebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:17.988172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b31dfee-8c28-41e2-8ce5-0d4baa1adebb admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:17.989114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b31dfee-8c28-41e2-8ce5-0d4baa1adebb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:17.990694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-9b31dfee-8c28-41e2-8ce5-0d4baa1adebb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:13:17.990694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Aug 25 05:13:17.990694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: Aug 25 05:13:18.000761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b31dfee-8c28-41e2-8ce5-0d4baa1adebb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.062877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 770/1548] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:17 2022] POST /identity/v3/users => generated 293 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:18.064071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d395c0b-1905-48b8-ad6f-ea420853ba5e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.064364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d395c0b-1905-48b8-ad6f-ea420853ba5e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.064642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d395c0b-1905-48b8-ad6f-ea420853ba5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.084079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8d395c0b-1905-48b8-ad6f-ea420853ba5e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.085354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8d395c0b-1905-48b8-ad6f-ea420853ba5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.093463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 771/1549] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.094662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02bba2-a813-437c-927f-118e4543aa8f None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.094947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02bba2-a813-437c-927f-118e4543aa8f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.095224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02bba2-a813-437c-927f-118e4543aa8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.112306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 779/1550] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:17 2022] POST /identity/v3/projects => generated 377 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:18.112306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a43e38-ddbf-4648-9094-fef545923d48 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.112306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a43e38-ddbf-4648-9094-fef545923d48 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.112306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a43e38-ddbf-4648-9094-fef545923d48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.116991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8d02bba2-a813-437c-927f-118e4543aa8f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.118315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8d02bba2-a813-437c-927f-118e4543aa8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.130354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f5a43e38-ddbf-4648-9094-fef545923d48 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.131735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f5a43e38-ddbf-4648-9094-fef545923d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.178107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 772/1551] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.179419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e522680-6bac-4c8b-b746-2f5154b97319 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.179775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e522680-6bac-4c8b-b746-2f5154b97319 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.180119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e522680-6bac-4c8b-b746-2f5154b97319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.197142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 780/1552] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.198499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3d22a-7718-4443-af05-20668821d3d2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.198785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3d22a-7718-4443-af05-20668821d3d2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.199063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3d22a-7718-4443-af05-20668821d3d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.200552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2e522680-6bac-4c8b-b746-2f5154b97319 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.201971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2e522680-6bac-4c8b-b746-2f5154b97319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.210631 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 773/1553] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.212083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83474d7f-953d-45e0-917a-fb672bc387d6 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.212375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83474d7f-953d-45e0-917a-fb672bc387d6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.212648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83474d7f-953d-45e0-917a-fb672bc387d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.220477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e9f3d22a-7718-4443-af05-20668821d3d2 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.221702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e9f3d22a-7718-4443-af05-20668821d3d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.229911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 781/1554] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.230597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f3d22a-7718-4443-af05-20668821d3d2 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.235024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-83474d7f-953d-45e0-917a-fb672bc387d6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.236258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-83474d7f-953d-45e0-917a-fb672bc387d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.240339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f3d22a-7718-4443-af05-20668821d3d2 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.244491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 774/1555] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.245772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998567ee-8085-4fb2-859f-b55e8f84d08b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.246058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998567ee-8085-4fb2-859f-b55e8f84d08b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.246301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5536a665-14b1-419e-8cce-2ef3550343af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.246457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998567ee-8085-4fb2-859f-b55e8f84d08b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.246922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5536a665-14b1-419e-8cce-2ef3550343af admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.247332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5536a665-14b1-419e-8cce-2ef3550343af admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.247619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5536a665-14b1-419e-8cce-2ef3550343af admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.248395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5536a665-14b1-419e-8cce-2ef3550343af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.250333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5536a665-14b1-419e-8cce-2ef3550343af admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.261361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 782/1556] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:18.264798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-998567ee-8085-4fb2-859f-b55e8f84d08b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.264963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5536a665-14b1-419e-8cce-2ef3550343af admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.266540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-998567ee-8085-4fb2-859f-b55e8f84d08b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.274437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 775/1557] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.274888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5536a665-14b1-419e-8cce-2ef3550343af admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.275595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c07e9c-58f4-4d1d-a387-d6473d566892 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.275873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c07e9c-58f4-4d1d-a387-d6473d566892 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.276146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c07e9c-58f4-4d1d-a387-d6473d566892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.279887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f089e433-1ce0-41a9-adba-2fea97220e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.280469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089e433-1ce0-41a9-adba-2fea97220e77 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.280751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089e433-1ce0-41a9-adba-2fea97220e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.281107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f089e433-1ce0-41a9-adba-2fea97220e77 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.281771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f089e433-1ce0-41a9-adba-2fea97220e77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.283681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f089e433-1ce0-41a9-adba-2fea97220e77 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.294403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-95c07e9c-58f4-4d1d-a387-d6473d566892 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.295581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-95c07e9c-58f4-4d1d-a387-d6473d566892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.303822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 776/1558] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.304981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56b018-d1e9-4a99-ac5a-a5f4763df5d1 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.305265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56b018-d1e9-4a99-ac5a-a5f4763df5d1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.305536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e56b018-d1e9-4a99-ac5a-a5f4763df5d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.324570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6e56b018-d1e9-4a99-ac5a-a5f4763df5d1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.325701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6e56b018-d1e9-4a99-ac5a-a5f4763df5d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.333397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 777/1559] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.334174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e56b018-d1e9-4a99-ac5a-a5f4763df5d1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.343240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e56b018-d1e9-4a99-ac5a-a5f4763df5d1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.347773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14a999-c1b3-4b0b-a072-793c9cfbf8de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.348333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14a999-c1b3-4b0b-a072-793c9cfbf8de admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.348614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14a999-c1b3-4b0b-a072-793c9cfbf8de admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.348885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14a999-c1b3-4b0b-a072-793c9cfbf8de admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.349506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14a999-c1b3-4b0b-a072-793c9cfbf8de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:18.349839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14a999-c1b3-4b0b-a072-793c9cfbf8de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.350849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WARNING py.warnings [None req-8a14a999-c1b3-4b0b-a072-793c9cfbf8de admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:13:18.350849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: warnings.warn(msg) Aug 25 05:13:18.350849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: Aug 25 05:13:18.351716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14a999-c1b3-4b0b-a072-793c9cfbf8de admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.360490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:18.363580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 778/1560] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:18.364681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e514492-cf5a-433e-b122-fb5a91fa6732 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.364964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e514492-cf5a-433e-b122-fb5a91fa6732 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.365236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e514492-cf5a-433e-b122-fb5a91fa6732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.377473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:18.386650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 783/1561] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/users => generated 289 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:18.386844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5e514492-cf5a-433e-b122-fb5a91fa6732 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.386970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f9b353-d69f-4c54-af7a-2c9b09b3908c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.386970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f9b353-d69f-4c54-af7a-2c9b09b3908c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.387123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5e514492-cf5a-433e-b122-fb5a91fa6732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.387200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f9b353-d69f-4c54-af7a-2c9b09b3908c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.393267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 779/1562] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.394123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e514492-cf5a-433e-b122-fb5a91fa6732 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.403715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e514492-cf5a-433e-b122-fb5a91fa6732 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.405167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e7f9b353-d69f-4c54-af7a-2c9b09b3908c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:18.406532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e7f9b353-d69f-4c54-af7a-2c9b09b3908c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:18.411436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82058e3-da74-4f6c-9bb1-4004e0216b16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.411436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82058e3-da74-4f6c-9bb1-4004e0216b16 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.411436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82058e3-da74-4f6c-9bb1-4004e0216b16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.411436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82058e3-da74-4f6c-9bb1-4004e0216b16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.411787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82058e3-da74-4f6c-9bb1-4004e0216b16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.411787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a82058e3-da74-4f6c-9bb1-4004e0216b16 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.414505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 784/1563] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:18.415269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f9b353-d69f-4c54-af7a-2c9b09b3908c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.422402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 780/1564] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:18.423082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82058e3-da74-4f6c-9bb1-4004e0216b16 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.427849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f9b353-d69f-4c54-af7a-2c9b09b3908c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.432664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82058e3-da74-4f6c-9bb1-4004e0216b16 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.433497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.434251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.434539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.434816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.435474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:18.435780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.438090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.440348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.440348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.440348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.440348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.440854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:18.441164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.442765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.450515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 785/1565] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:18.451227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.460820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a84bd9-8871-4676-8db6-658342d16564 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.461071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 781/1566] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:18.461823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.470927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.471532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.471637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5033ce02-63c4-4bd3-a77c-f0b86da1135e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.471822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.472121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.472793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:18.473099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.474728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.476188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74d6e66-718c-4ed9-9fad-cfaad0bc44e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.476768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74d6e66-718c-4ed9-9fad-cfaad0bc44e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.477045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74d6e66-718c-4ed9-9fad-cfaad0bc44e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.477348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74d6e66-718c-4ed9-9fad-cfaad0bc44e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.477984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74d6e66-718c-4ed9-9fad-cfaad0bc44e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.479852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74d6e66-718c-4ed9-9fad-cfaad0bc44e0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.487998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 786/1567] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:18.488945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.499725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb621ca7-1489-4959-ba56-5adeb175c6cd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.505075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b5e57-ec6b-4d7b-b079-f59d54ea1232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.505729 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b5e57-ec6b-4d7b-b079-f59d54ea1232 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.506012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b5e57-ec6b-4d7b-b079-f59d54ea1232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.506294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b5e57-ec6b-4d7b-b079-f59d54ea1232 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.506997 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2b5e57-ec6b-4d7b-b079-f59d54ea1232 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.508584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2b5e57-ec6b-4d7b-b079-f59d54ea1232 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.518810 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 787/1568] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:18.519592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b5e57-ec6b-4d7b-b079-f59d54ea1232 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.528667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b5e57-ec6b-4d7b-b079-f59d54ea1232 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.533768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.534313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.534603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.534902 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.535513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:18.535815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.537322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.549128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 788/1569] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:18.549927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.558912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0002a15e-581a-4c74-8d68-027a1ce8c65b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.563483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.564032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.564311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.564623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.565234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:18.565536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.567085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.581664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 789/1570] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:18.581664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.586731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 782/1571] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/projects => generated 389 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:18.586731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74d6e66-718c-4ed9-9fad-cfaad0bc44e0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.592613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f10cef3-6518-429c-85ae-2eb6483788c8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.646525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74d6e66-718c-4ed9-9fad-cfaad0bc44e0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.651010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.651152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd07bb-6b85-42ac-9f2d-d67ec15ca6ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.651687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.651790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd07bb-6b85-42ac-9f2d-d67ec15ca6ae admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.651974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.652094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd07bb-6b85-42ac-9f2d-d67ec15ca6ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.652245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd07bb-6b85-42ac-9f2d-d67ec15ca6ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.652463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] PATH_INFO: `/v3/projects/f640cb592ed1455a827976752ce2f49d/users/ec39d995899c4d43bb4f8ffd9e6d2ef3/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.652862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd07bb-6b85-42ac-9f2d-d67ec15ca6ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.654783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd07bb-6b85-42ac-9f2d-d67ec15ca6ae admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.667779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] RBAC: Authorizing `identity:create_grant(project_id=f640cb592ed1455a827976752ce2f49d, user_id=ec39d995899c4d43bb4f8ffd9e6d2ef3, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.668802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:13:18.668802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Aug 25 05:13:18.668802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: Aug 25 05:13:18.670762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.742959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 783/1572] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/projects => generated 371 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:18.743757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd07bb-6b85-42ac-9f2d-d67ec15ca6ae admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.759061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 790/1573] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:18 2022] PUT /identity/v3/projects/f640cb592ed1455a827976752ce2f49d/users/ec39d995899c4d43bb4f8ffd9e6d2ef3/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:18.761227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.806513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd07bb-6b85-42ac-9f2d-d67ec15ca6ae admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.822297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee4a956-fb5f-4f46-93be-7d43530adeaf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.826891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52de232-39e4-4aa8-8561-c818c1207e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.827489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52de232-39e4-4aa8-8561-c818c1207e5e admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.827795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52de232-39e4-4aa8-8561-c818c1207e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.828069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52de232-39e4-4aa8-8561-c818c1207e5e admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.828687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52de232-39e4-4aa8-8561-c818c1207e5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.830631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52de232-39e4-4aa8-8561-c818c1207e5e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.843063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:18.846615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:18.861347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d17bf42-8d20-48f4-95c6-d45413860d82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.862001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d17bf42-8d20-48f4-95c6-d45413860d82 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.862281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d17bf42-8d20-48f4-95c6-d45413860d82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.862552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d17bf42-8d20-48f4-95c6-d45413860d82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.863168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d17bf42-8d20-48f4-95c6-d45413860d82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.865058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d17bf42-8d20-48f4-95c6-d45413860d82 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:18.920456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 791/1574] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/projects => generated 377 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:18.921287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52de232-39e4-4aa8-8561-c818c1207e5e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:18.982173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52de232-39e4-4aa8-8561-c818c1207e5e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:18.987286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741b5533-2c77-4a01-8360-e321468773e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:18.987885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741b5533-2c77-4a01-8360-e321468773e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:18.988166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741b5533-2c77-4a01-8360-e321468773e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:18.988437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741b5533-2c77-4a01-8360-e321468773e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:18.989074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741b5533-2c77-4a01-8360-e321468773e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:18.990696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741b5533-2c77-4a01-8360-e321468773e3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.000667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 792/1575] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:18 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.001565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741b5533-2c77-4a01-8360-e321468773e3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.011046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741b5533-2c77-4a01-8360-e321468773e3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.019910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 784/1576] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:18 2022] POST /identity/v3/projects => generated 403 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.019910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d17bf42-8d20-48f4-95c6-d45413860d82 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.070226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765f9954-7079-4fd7-9390-baf7289cc491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.070841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765f9954-7079-4fd7-9390-baf7289cc491 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.071117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765f9954-7079-4fd7-9390-baf7289cc491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.071419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765f9954-7079-4fd7-9390-baf7289cc491 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.072094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765f9954-7079-4fd7-9390-baf7289cc491 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.074099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765f9954-7079-4fd7-9390-baf7289cc491 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.074854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d17bf42-8d20-48f4-95c6-d45413860d82 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.079545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdcc1f3-545f-48bf-bc84-24469e39fe03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.080142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdcc1f3-545f-48bf-bc84-24469e39fe03 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.080417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdcc1f3-545f-48bf-bc84-24469e39fe03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.080685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdcc1f3-545f-48bf-bc84-24469e39fe03 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.081297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdcc1f3-545f-48bf-bc84-24469e39fe03 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.083248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdcc1f3-545f-48bf-bc84-24469e39fe03 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.189229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 793/1577] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/projects => generated 365 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.190315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765f9954-7079-4fd7-9390-baf7289cc491 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.200324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 785/1578] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/users => generated 295 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.200324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdcc1f3-545f-48bf-bc84-24469e39fe03 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.210895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdcc1f3-545f-48bf-bc84-24469e39fe03 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.256909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765f9954-7079-4fd7-9390-baf7289cc491 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.261911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f82daa-429a-4b79-a35a-fcf2b78454cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.262727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f82daa-429a-4b79-a35a-fcf2b78454cf admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.263008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f82daa-429a-4b79-a35a-fcf2b78454cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.263281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f82daa-429a-4b79-a35a-fcf2b78454cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.263948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f82daa-429a-4b79-a35a-fcf2b78454cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.266204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f82daa-429a-4b79-a35a-fcf2b78454cf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.272576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5914cc57-ede3-495c-a29f-9597d1617bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.273200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5914cc57-ede3-495c-a29f-9597d1617bd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.273480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5914cc57-ede3-495c-a29f-9597d1617bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.273781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5914cc57-ede3-495c-a29f-9597d1617bd2 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.274455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5914cc57-ede3-495c-a29f-9597d1617bd2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.276559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5914cc57-ede3-495c-a29f-9597d1617bd2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.356043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 786/1579] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.356244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 794/1580] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/projects => generated 393 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.356332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a26db46-017e-4851-b59e-77756e51aa58 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.356332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a26db46-017e-4851-b59e-77756e51aa58 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.356332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a26db46-017e-4851-b59e-77756e51aa58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.356595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f82daa-429a-4b79-a35a-fcf2b78454cf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.360260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:19.375086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5a26db46-017e-4851-b59e-77756e51aa58 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:19.376285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5a26db46-017e-4851-b59e-77756e51aa58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:19.417674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:19.418730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f82daa-429a-4b79-a35a-fcf2b78454cf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.424336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041e1b12-5e31-4d0e-97ec-a6acf1b0a1dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.428442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041e1b12-5e31-4d0e-97ec-a6acf1b0a1dd admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.428442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041e1b12-5e31-4d0e-97ec-a6acf1b0a1dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.428442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041e1b12-5e31-4d0e-97ec-a6acf1b0a1dd admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.428442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041e1b12-5e31-4d0e-97ec-a6acf1b0a1dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.428442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041e1b12-5e31-4d0e-97ec-a6acf1b0a1dd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.436882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 787/1581] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:19.437718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a26db46-017e-4851-b59e-77756e51aa58 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.447152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a26db46-017e-4851-b59e-77756e51aa58 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.451765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64ba2b9-8213-44a6-bf15-4c3b1e58a8df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.452414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ba2b9-8213-44a6-bf15-4c3b1e58a8df admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.452696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ba2b9-8213-44a6-bf15-4c3b1e58a8df admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.452968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ba2b9-8213-44a6-bf15-4c3b1e58a8df admin admin] PATH_INFO: `/v3/projects/e1f4463eb245468984f4747372d53fc6/users/3107aef7eb954ac4845c8903b005ce93/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.467180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64ba2b9-8213-44a6-bf15-4c3b1e58a8df admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1f4463eb245468984f4747372d53fc6, user_id=3107aef7eb954ac4845c8903b005ce93, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.470373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64ba2b9-8213-44a6-bf15-4c3b1e58a8df admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.511421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 795/1582] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/users => generated 289 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.512378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041e1b12-5e31-4d0e-97ec-a6acf1b0a1dd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.521718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041e1b12-5e31-4d0e-97ec-a6acf1b0a1dd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.526248 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e6517-ce6b-40e6-9c39-1e5ff051f3fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.526812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e6517-ce6b-40e6-9c39-1e5ff051f3fe admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.527092 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e6517-ce6b-40e6-9c39-1e5ff051f3fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.527362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1e6517-ce6b-40e6-9c39-1e5ff051f3fe admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.527964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e6517-ce6b-40e6-9c39-1e5ff051f3fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.529882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1e6517-ce6b-40e6-9c39-1e5ff051f3fe admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.551594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 788/1583] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:19 2022] PUT /identity/v3/projects/e1f4463eb245468984f4747372d53fc6/users/3107aef7eb954ac4845c8903b005ce93/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:19.552510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64ba2b9-8213-44a6-bf15-4c3b1e58a8df admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.612663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64ba2b9-8213-44a6-bf15-4c3b1e58a8df admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.617253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368c950-c39b-4a36-a223-388335b3babe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.617938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368c950-c39b-4a36-a223-388335b3babe admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.618220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368c950-c39b-4a36-a223-388335b3babe admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.618516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368c950-c39b-4a36-a223-388335b3babe admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.619133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f368c950-c39b-4a36-a223-388335b3babe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.621047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f368c950-c39b-4a36-a223-388335b3babe admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.629286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 796/1584] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/users => generated 302 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.629286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e6517-ce6b-40e6-9c39-1e5ff051f3fe admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.637792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1e6517-ce6b-40e6-9c39-1e5ff051f3fe admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.642342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc95521-c432-446c-b636-e55600ac47be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.642965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc95521-c432-446c-b636-e55600ac47be admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.643245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc95521-c432-446c-b636-e55600ac47be admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.643518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc95521-c432-446c-b636-e55600ac47be admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.644154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc95521-c432-446c-b636-e55600ac47be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.645763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc95521-c432-446c-b636-e55600ac47be admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.655519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 797/1585] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.656855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc95521-c432-446c-b636-e55600ac47be admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.666046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc95521-c432-446c-b636-e55600ac47be admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.670519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1289dc34-96b5-4a7c-9dcb-addd1c4808fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.671075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289dc34-96b5-4a7c-9dcb-addd1c4808fd admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.671390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289dc34-96b5-4a7c-9dcb-addd1c4808fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.672339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289dc34-96b5-4a7c-9dcb-addd1c4808fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.672942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1289dc34-96b5-4a7c-9dcb-addd1c4808fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.674483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1289dc34-96b5-4a7c-9dcb-addd1c4808fd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.685636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 798/1586] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.686520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1289dc34-96b5-4a7c-9dcb-addd1c4808fd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.695894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1289dc34-96b5-4a7c-9dcb-addd1c4808fd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.700328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa63d41-d452-4264-8617-5898f8b24b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.700903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa63d41-d452-4264-8617-5898f8b24b56 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.701186 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa63d41-d452-4264-8617-5898f8b24b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.701490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa63d41-d452-4264-8617-5898f8b24b56 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.702125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa63d41-d452-4264-8617-5898f8b24b56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.704542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa63d41-d452-4264-8617-5898f8b24b56 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.710107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 789/1587] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/users => generated 283 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.710803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368c950-c39b-4a36-a223-388335b3babe admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.765055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368c950-c39b-4a36-a223-388335b3babe admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.769170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a492280f-7752-41c7-a75f-090b8df03c78 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.769805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a492280f-7752-41c7-a75f-090b8df03c78 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.770088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a492280f-7752-41c7-a75f-090b8df03c78 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.770362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a492280f-7752-41c7-a75f-090b8df03c78 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.779342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a492280f-7752-41c7-a75f-090b8df03c78 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.780921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a492280f-7752-41c7-a75f-090b8df03c78 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.786444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 799/1588] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/users => generated 297 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:19.790430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa63d41-d452-4264-8617-5898f8b24b56 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.791574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 790/1589] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:19.792835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b3c67e-ac7a-42d5-9713-9b00db9ef602 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.793064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b3c67e-ac7a-42d5-9713-9b00db9ef602 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.793340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b3c67e-ac7a-42d5-9713-9b00db9ef602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.800121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa63d41-d452-4264-8617-5898f8b24b56 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.804760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d2a5be-e07d-4285-aa82-6e6bac29db98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.805360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2a5be-e07d-4285-aa82-6e6bac29db98 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.805667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2a5be-e07d-4285-aa82-6e6bac29db98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.805943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2a5be-e07d-4285-aa82-6e6bac29db98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.806584 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d2a5be-e07d-4285-aa82-6e6bac29db98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.808326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d2a5be-e07d-4285-aa82-6e6bac29db98 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.815914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-62b3c67e-ac7a-42d5-9713-9b00db9ef602 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:19.817124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-62b3c67e-ac7a-42d5-9713-9b00db9ef602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:19.821471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 800/1590] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.821471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d2a5be-e07d-4285-aa82-6e6bac29db98 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.829302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 791/1591] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:19.836585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b3c67e-ac7a-42d5-9713-9b00db9ef602 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.836698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d2a5be-e07d-4285-aa82-6e6bac29db98 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.837487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff7e1f-5cba-4752-a700-f4ff5d7d23dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.838119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff7e1f-5cba-4752-a700-f4ff5d7d23dd admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.838409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff7e1f-5cba-4752-a700-f4ff5d7d23dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.838664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff7e1f-5cba-4752-a700-f4ff5d7d23dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.839552 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfff7e1f-5cba-4752-a700-f4ff5d7d23dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.840698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b3c67e-ac7a-42d5-9713-9b00db9ef602 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.841213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfff7e1f-5cba-4752-a700-f4ff5d7d23dd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.846291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8833d5f-8096-44cf-82f3-24c20f2a1eb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.846940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8833d5f-8096-44cf-82f3-24c20f2a1eb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.847222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8833d5f-8096-44cf-82f3-24c20f2a1eb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.847489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8833d5f-8096-44cf-82f3-24c20f2a1eb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.848155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8833d5f-8096-44cf-82f3-24c20f2a1eb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.849796 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8833d5f-8096-44cf-82f3-24c20f2a1eb0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.855817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 801/1592] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.855817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff7e1f-5cba-4752-a700-f4ff5d7d23dd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.861396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 792/1593] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.866846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8833d5f-8096-44cf-82f3-24c20f2a1eb0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.866958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff7e1f-5cba-4752-a700-f4ff5d7d23dd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.868702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cfb7f8-6f12-4593-a8a2-acbc89d4e7b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.869357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfb7f8-6f12-4593-a8a2-acbc89d4e7b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.869674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfb7f8-6f12-4593-a8a2-acbc89d4e7b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.869948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfb7f8-6f12-4593-a8a2-acbc89d4e7b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.870611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cfb7f8-6f12-4593-a8a2-acbc89d4e7b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.872213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cfb7f8-6f12-4593-a8a2-acbc89d4e7b4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.872338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8833d5f-8096-44cf-82f3-24c20f2a1eb0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.876845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf634d-caa5-47aa-a3ee-4a6ac00ad47e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.877424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf634d-caa5-47aa-a3ee-4a6ac00ad47e admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.877745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf634d-caa5-47aa-a3ee-4a6ac00ad47e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.878002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf634d-caa5-47aa-a3ee-4a6ac00ad47e admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.878623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cf634d-caa5-47aa-a3ee-4a6ac00ad47e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.880194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cf634d-caa5-47aa-a3ee-4a6ac00ad47e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.882692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 802/1594] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.883410 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cfb7f8-6f12-4593-a8a2-acbc89d4e7b4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.891792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 793/1595] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.898462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf634d-caa5-47aa-a3ee-4a6ac00ad47e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.898572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cfb7f8-6f12-4593-a8a2-acbc89d4e7b4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.899492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9148d8-e399-4297-95bd-8fe7e8fafbd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.900378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9148d8-e399-4297-95bd-8fe7e8fafbd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.900691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9148d8-e399-4297-95bd-8fe7e8fafbd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.900969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9148d8-e399-4297-95bd-8fe7e8fafbd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.901626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9148d8-e399-4297-95bd-8fe7e8fafbd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.902993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf634d-caa5-47aa-a3ee-4a6ac00ad47e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.903228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9148d8-e399-4297-95bd-8fe7e8fafbd8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.908324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5686db-bcfd-4cb0-afb5-976d0abb2a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.908916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5686db-bcfd-4cb0-afb5-976d0abb2a57 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.909193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5686db-bcfd-4cb0-afb5-976d0abb2a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.909486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5686db-bcfd-4cb0-afb5-976d0abb2a57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.910251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5686db-bcfd-4cb0-afb5-976d0abb2a57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.911814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5686db-bcfd-4cb0-afb5-976d0abb2a57 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.914055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 803/1596] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.917226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9148d8-e399-4297-95bd-8fe7e8fafbd8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.926465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 794/1597] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.926465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5686db-bcfd-4cb0-afb5-976d0abb2a57 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.926723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9148d8-e399-4297-95bd-8fe7e8fafbd8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.930616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5988a2fe-91c0-432b-9978-23a7419dbac1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.931407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5988a2fe-91c0-432b-9978-23a7419dbac1 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.931911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5988a2fe-91c0-432b-9978-23a7419dbac1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.932126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5988a2fe-91c0-432b-9978-23a7419dbac1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.933929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5686db-bcfd-4cb0-afb5-976d0abb2a57 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.938668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f96f7b-2d66-45cd-accf-8ff6b3f15cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.939316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f96f7b-2d66-45cd-accf-8ff6b3f15cb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.944290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f96f7b-2d66-45cd-accf-8ff6b3f15cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.944290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f96f7b-2d66-45cd-accf-8ff6b3f15cb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.944290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f96f7b-2d66-45cd-accf-8ff6b3f15cb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.944679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5988a2fe-91c0-432b-9978-23a7419dbac1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.944783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f96f7b-2d66-45cd-accf-8ff6b3f15cb5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.944871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5988a2fe-91c0-432b-9978-23a7419dbac1 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:19.954487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 804/1598] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:19.958312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 795/1599] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:19.958438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5988a2fe-91c0-432b-9978-23a7419dbac1 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.958542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f96f7b-2d66-45cd-accf-8ff6b3f15cb5 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:19.964696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:19.967813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5988a2fe-91c0-432b-9978-23a7419dbac1 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.972448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f96f7b-2d66-45cd-accf-8ff6b3f15cb5 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:19.978876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a905e5-296a-4e10-9913-3b2a6b254512 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.978876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a905e5-296a-4e10-9913-3b2a6b254512 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.978876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a905e5-296a-4e10-9913-3b2a6b254512 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.978876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a905e5-296a-4e10-9913-3b2a6b254512 admin admin] PATH_INFO: `/v3/projects/dcd7d2bdaffb4d168f27cf6444e005dd/users/3326fcf3ccbc4a978e9e52e25456ad98/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.979243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18b2a84-1f16-4c11-a47d-3b051def1d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:19.980604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b2a84-1f16-4c11-a47d-3b051def1d9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:19.980887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b2a84-1f16-4c11-a47d-3b051def1d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:19.981299 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18b2a84-1f16-4c11-a47d-3b051def1d9e admin admin] PATH_INFO: `/v3/projects/4b9739a947184e05b94cfd3b305bd9f5/users/1f0948bf60d44b8d8c26738d6339b058/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:19.991140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a905e5-296a-4e10-9913-3b2a6b254512 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dcd7d2bdaffb4d168f27cf6444e005dd, user_id=3326fcf3ccbc4a978e9e52e25456ad98, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:19.993937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a905e5-296a-4e10-9913-3b2a6b254512 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.001253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18b2a84-1f16-4c11-a47d-3b051def1d9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b9739a947184e05b94cfd3b305bd9f5, user_id=1f0948bf60d44b8d8c26738d6339b058, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:20.004007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18b2a84-1f16-4c11-a47d-3b051def1d9e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.103379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 805/1600] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:19 2022] PUT /identity/v3/projects/dcd7d2bdaffb4d168f27cf6444e005dd/users/3326fcf3ccbc4a978e9e52e25456ad98/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:20.113610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a905e5-296a-4e10-9913-3b2a6b254512 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:20.114282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 796/1601] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:19 2022] PUT /identity/v3/projects/4b9739a947184e05b94cfd3b305bd9f5/users/1f0948bf60d44b8d8c26738d6339b058/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:20.115343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18b2a84-1f16-4c11-a47d-3b051def1d9e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:20.129834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:20.183501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a905e5-296a-4e10-9913-3b2a6b254512 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:20.189654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a0f20e-4f65-43a5-8efd-5abd22a18c33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:20.190480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a0f20e-4f65-43a5-8efd-5abd22a18c33 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.191079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18b2a84-1f16-4c11-a47d-3b051def1d9e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:20.191227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a0f20e-4f65-43a5-8efd-5abd22a18c33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.191227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a0f20e-4f65-43a5-8efd-5abd22a18c33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.192118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a0f20e-4f65-43a5-8efd-5abd22a18c33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:20.194106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a0f20e-4f65-43a5-8efd-5abd22a18c33 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.198338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd80cf0-37fa-40ca-9d2b-a00a87c2516d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:20.198974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd80cf0-37fa-40ca-9d2b-a00a87c2516d admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.199323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd80cf0-37fa-40ca-9d2b-a00a87c2516d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.199665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd80cf0-37fa-40ca-9d2b-a00a87c2516d admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.201052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd80cf0-37fa-40ca-9d2b-a00a87c2516d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:20.202805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd80cf0-37fa-40ca-9d2b-a00a87c2516d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.209736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 806/1602] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:20 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:20.209736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a0f20e-4f65-43a5-8efd-5abd22a18c33 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:20.216010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 797/1603] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:20 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:20.216789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd80cf0-37fa-40ca-9d2b-a00a87c2516d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:20.218056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a0f20e-4f65-43a5-8efd-5abd22a18c33 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:20.225971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9db01b-ded6-4e43-9034-610928629d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:20.226639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9db01b-ded6-4e43-9034-610928629d52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.228008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9db01b-ded6-4e43-9034-610928629d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.228300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9db01b-ded6-4e43-9034-610928629d52 admin admin] PATH_INFO: `/v3/projects/2eb9cdd7b67546ca8eb089247f0a0ce1/users/d61cf1670a7a43519c7258cd467227b6/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.231567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd80cf0-37fa-40ca-9d2b-a00a87c2516d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:20.239241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae64d8f7-f830-4101-bd57-276510adf2bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:20.239914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae64d8f7-f830-4101-bd57-276510adf2bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.240317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae64d8f7-f830-4101-bd57-276510adf2bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.240560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae64d8f7-f830-4101-bd57-276510adf2bb admin admin] PATH_INFO: `/v3/projects/4caa2607a9024662846dbce3fa4907db/users/f112c0816b414fca88217af02a6f3f9c/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.249503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9db01b-ded6-4e43-9034-610928629d52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2eb9cdd7b67546ca8eb089247f0a0ce1, user_id=d61cf1670a7a43519c7258cd467227b6, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:20.253206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9db01b-ded6-4e43-9034-610928629d52 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.257465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae64d8f7-f830-4101-bd57-276510adf2bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=4caa2607a9024662846dbce3fa4907db, user_id=f112c0816b414fca88217af02a6f3f9c, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:20.260271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae64d8f7-f830-4101-bd57-276510adf2bb admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.312227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 807/1604] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:20 2022] PUT /identity/v3/projects/2eb9cdd7b67546ca8eb089247f0a0ce1/users/d61cf1670a7a43519c7258cd467227b6/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:20.313814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e1ae4-b4a5-41e0-beeb-40e2516e9557 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.314027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e1ae4-b4a5-41e0-beeb-40e2516e9557 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.314310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e1ae4-b4a5-41e0-beeb-40e2516e9557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.333149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5c6e1ae4-b4a5-41e0-beeb-40e2516e9557 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:20.334703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5c6e1ae4-b4a5-41e0-beeb-40e2516e9557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:20.357561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 798/1605] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:20 2022] PUT /identity/v3/projects/4caa2607a9024662846dbce3fa4907db/users/f112c0816b414fca88217af02a6f3f9c/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:20.358838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a466e0-6341-4eba-80ed-ec7167ee38d9 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.359865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a466e0-6341-4eba-80ed-ec7167ee38d9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.360339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a466e0-6341-4eba-80ed-ec7167ee38d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.394298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-45a466e0-6341-4eba-80ed-ec7167ee38d9 None None] MFA Rules not processed for user `1f0948bf60d44b8d8c26738d6339b058`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:20.395532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-45a466e0-6341-4eba-80ed-ec7167ee38d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:20.429274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 808/1606] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:20.430845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6e1ae4-b4a5-41e0-beeb-40e2516e9557 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:20.447782 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6e1ae4-b4a5-41e0-beeb-40e2516e9557 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:20.479369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 799/1607] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:20 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:20.485036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a466e0-6341-4eba-80ed-ec7167ee38d9 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:20.541336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ad0222-60b7-40e9-9984-6133a7ab06e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:20.542060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ad0222-60b7-40e9-9984-6133a7ab06e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.542342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ad0222-60b7-40e9-9984-6133a7ab06e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.543163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ad0222-60b7-40e9-9984-6133a7ab06e7 admin admin] PATH_INFO: `/v3/projects/99e2bf92350a413a8f593770a3abe5d2/users/bc62fa0058a340f882759c14873a0be5/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.544083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:20.564223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ad0222-60b7-40e9-9984-6133a7ab06e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=99e2bf92350a413a8f593770a3abe5d2, user_id=bc62fa0058a340f882759c14873a0be5, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:20.568279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ad0222-60b7-40e9-9984-6133a7ab06e7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.569053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a466e0-6341-4eba-80ed-ec7167ee38d9 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:20.575933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:20.576640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.576973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.577251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] PATH_INFO: `/v3/projects/8afac3f36b8449aa8a25a376aa2c3ed9/users/b6ca9962541744bf9d67cc3e05721413/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.594566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8afac3f36b8449aa8a25a376aa2c3ed9, user_id=b6ca9962541744bf9d67cc3e05721413, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:20.597207 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.679223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 809/1608] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:20 2022] PUT /identity/v3/projects/99e2bf92350a413a8f593770a3abe5d2/users/bc62fa0058a340f882759c14873a0be5/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:20.680528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682e373-9d90-474e-971d-76cd079c588f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.680812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682e373-9d90-474e-971d-76cd079c588f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.681087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b682e373-9d90-474e-971d-76cd079c588f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.699897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b682e373-9d90-474e-971d-76cd079c588f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:20.701095 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b682e373-9d90-474e-971d-76cd079c588f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:20.705626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:20.760926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 810/1609] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:20.762275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ed21d-e0de-492e-97c8-6f407387940f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.762564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ed21d-e0de-492e-97c8-6f407387940f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.762842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ed21d-e0de-492e-97c8-6f407387940f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.776315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 800/1610] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:20 2022] PUT /identity/v3/projects/8afac3f36b8449aa8a25a376aa2c3ed9/users/b6ca9962541744bf9d67cc3e05721413/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:20.777813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:20.780525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:20.803184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7c5ed21d-e0de-492e-97c8-6f407387940f None None] MFA Rules not processed for user `f112c0816b414fca88217af02a6f3f9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:20.803859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7c5ed21d-e0de-492e-97c8-6f407387940f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:20.845014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d812c5-a4b9-4b0f-aa49-bb077c208e37 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:20.850095 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08142cb4-fcc8-4792-8e35-17bc2cb03e3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:20.850775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08142cb4-fcc8-4792-8e35-17bc2cb03e3d service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.851066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08142cb4-fcc8-4792-8e35-17bc2cb03e3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.851479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08142cb4-fcc8-4792-8e35-17bc2cb03e3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.900924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 811/1611] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:20 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:20.902557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6076046-ba5f-4f29-95c0-9eeb0db35ee6 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.903065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6076046-ba5f-4f29-95c0-9eeb0db35ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.903581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6076046-ba5f-4f29-95c0-9eeb0db35ee6 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.906780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 812/1612] 10.222.0.16 () {54 vars in 962 bytes} [Thu Aug 25 05:13:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:13:20.907519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ef6ca-e107-4218-b679-c139ed396333 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.907884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ef6ca-e107-4218-b679-c139ed396333 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.908232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ef6ca-e107-4218-b679-c139ed396333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:20.947900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d64ef6ca-e107-4218-b679-c139ed396333 None None] MFA Rules not processed for user `bc62fa0058a340f882759c14873a0be5`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:20.949949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d64ef6ca-e107-4218-b679-c139ed396333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:20.960832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08142cb4-fcc8-4792-8e35-17bc2cb03e3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:20.962469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08142cb4-fcc8-4792-8e35-17bc2cb03e3d service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:20.974347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 801/1613] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:20.978584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08142cb4-fcc8-4792-8e35-17bc2cb03e3d service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:20.988118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08142cb4-fcc8-4792-8e35-17bc2cb03e3d service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:20.991992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931b6493-1ce8-43e7-9119-4ca7d0fecb06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:20.992584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931b6493-1ce8-43e7-9119-4ca7d0fecb06 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:20.992867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931b6493-1ce8-43e7-9119-4ca7d0fecb06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:20.993136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931b6493-1ce8-43e7-9119-4ca7d0fecb06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.003035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931b6493-1ce8-43e7-9119-4ca7d0fecb06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.004480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931b6493-1ce8-43e7-9119-4ca7d0fecb06 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.014225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 802/1614] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:21.015977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162e928d-1401-4736-b3aa-a4c89811fc8c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.016267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162e928d-1401-4736-b3aa-a4c89811fc8c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.016583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162e928d-1401-4736-b3aa-a4c89811fc8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.024087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 813/1615] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:20 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:21.025041 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64ef6ca-e107-4218-b679-c139ed396333 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.037216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-162e928d-1401-4736-b3aa-a4c89811fc8c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:21.038739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-162e928d-1401-4736-b3aa-a4c89811fc8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:21.039029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64ef6ca-e107-4218-b679-c139ed396333 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.043738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc226c12-d09d-4555-ac56-46cb1ae46e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.044384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc226c12-d09d-4555-ac56-46cb1ae46e95 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.044616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc226c12-d09d-4555-ac56-46cb1ae46e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.044890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc226c12-d09d-4555-ac56-46cb1ae46e95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.045689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc226c12-d09d-4555-ac56-46cb1ae46e95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.047451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 803/1616] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:21.047748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc226c12-d09d-4555-ac56-46cb1ae46e95 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.048520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dbd7ec-f3b2-461f-99a7-85f360302c6c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.048811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dbd7ec-f3b2-461f-99a7-85f360302c6c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.049088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dbd7ec-f3b2-461f-99a7-85f360302c6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.102757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-14dbd7ec-f3b2-461f-99a7-85f360302c6c None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:21.104538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-14dbd7ec-f3b2-461f-99a7-85f360302c6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:21.160872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:21.167918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 814/1617] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:21 2022] POST /identity/v3/projects => generated 403 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:21.167918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc226c12-d09d-4555-ac56-46cb1ae46e95 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.240680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 804/1618] 10.222.0.16 () {58 vars in 1059 bytes} [Thu Aug 25 05:13:21 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:21.241806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dbd7ec-f3b2-461f-99a7-85f360302c6c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.250898 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc226c12-d09d-4555-ac56-46cb1ae46e95 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.254847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0124cbc-881a-465f-9a12-5143aaf9b4cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.255462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0124cbc-881a-465f-9a12-5143aaf9b4cf service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.255747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0124cbc-881a-465f-9a12-5143aaf9b4cf service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.256050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0124cbc-881a-465f-9a12-5143aaf9b4cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.260247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dbd7ec-f3b2-461f-99a7-85f360302c6c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.332104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db3c243-504e-4850-9916-1b6c24aa1b26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.332732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3c243-504e-4850-9916-1b6c24aa1b26 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.333119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3c243-504e-4850-9916-1b6c24aa1b26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.333401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db3c243-504e-4850-9916-1b6c24aa1b26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.350830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0124cbc-881a-465f-9a12-5143aaf9b4cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.352398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0124cbc-881a-465f-9a12-5143aaf9b4cf service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.358387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:21.363599 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 815/1619] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:21 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:21.364457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0124cbc-881a-465f-9a12-5143aaf9b4cf service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.402943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db3c243-504e-4850-9916-1b6c24aa1b26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.404381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db3c243-504e-4850-9916-1b6c24aa1b26 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.418421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 805/1620] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:21.419500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db3c243-504e-4850-9916-1b6c24aa1b26 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.432313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0124cbc-881a-465f-9a12-5143aaf9b4cf service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.438756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f300aa2-3b4c-4cbe-a3d0-47d683127834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.441212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f300aa2-3b4c-4cbe-a3d0-47d683127834 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.441445 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f300aa2-3b4c-4cbe-a3d0-47d683127834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.441803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f300aa2-3b4c-4cbe-a3d0-47d683127834 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.442528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f300aa2-3b4c-4cbe-a3d0-47d683127834 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.445554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f300aa2-3b4c-4cbe-a3d0-47d683127834 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.508340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db3c243-504e-4850-9916-1b6c24aa1b26 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.514777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bda6e69-113d-4e11-a826-c875b6e35b05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.516113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda6e69-113d-4e11-a826-c875b6e35b05 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.516461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda6e69-113d-4e11-a826-c875b6e35b05 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.516726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bda6e69-113d-4e11-a826-c875b6e35b05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.550885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 816/1621] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:21 2022] POST /identity/v3/users => generated 308 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:21.554947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f300aa2-3b4c-4cbe-a3d0-47d683127834 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.577313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f300aa2-3b4c-4cbe-a3d0-47d683127834 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.592326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e096e5b3-f891-4743-82fd-8ddbb60219cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.593178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e096e5b3-f891-4743-82fd-8ddbb60219cc admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.594082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e096e5b3-f891-4743-82fd-8ddbb60219cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.596063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e096e5b3-f891-4743-82fd-8ddbb60219cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.596474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e096e5b3-f891-4743-82fd-8ddbb60219cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.602616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e096e5b3-f891-4743-82fd-8ddbb60219cc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.629061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 817/1622] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:21 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:21.633307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e096e5b3-f891-4743-82fd-8ddbb60219cc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.638202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bda6e69-113d-4e11-a826-c875b6e35b05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.641229 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bda6e69-113d-4e11-a826-c875b6e35b05 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.655299 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 806/1623] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:21 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:21.662252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e096e5b3-f891-4743-82fd-8ddbb60219cc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.673500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0174e-68cb-4bf7-8f2d-20895225c08d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.674521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0174e-68cb-4bf7-8f2d-20895225c08d admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.674822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0174e-68cb-4bf7-8f2d-20895225c08d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.675223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0174e-68cb-4bf7-8f2d-20895225c08d admin admin] PATH_INFO: `/v3/projects/00f3432914874e9699e2c7b9b7917e44/users/a22fcf5c814a452bace976751ceb10d7/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.702555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f0174e-68cb-4bf7-8f2d-20895225c08d admin admin] RBAC: Authorizing `identity:create_grant(project_id=00f3432914874e9699e2c7b9b7917e44, user_id=a22fcf5c814a452bace976751ceb10d7, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.702555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f0174e-68cb-4bf7-8f2d-20895225c08d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.782677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 818/1624] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:21 2022] PUT /identity/v3/projects/00f3432914874e9699e2c7b9b7917e44/users/a22fcf5c814a452bace976751ceb10d7/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:13:21.786783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bda6e69-113d-4e11-a826-c875b6e35b05 service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.804397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bda6e69-113d-4e11-a826-c875b6e35b05 service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.834155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:21.914144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d6ba94-efa5-4190-80b1-24992b1c7b2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.917077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6ba94-efa5-4190-80b1-24992b1c7b2c admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.917077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6ba94-efa5-4190-80b1-24992b1c7b2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.917077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d6ba94-efa5-4190-80b1-24992b1c7b2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.917077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d6ba94-efa5-4190-80b1-24992b1c7b2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.917511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d6ba94-efa5-4190-80b1-24992b1c7b2c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.936612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 807/1625] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:21 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:21.938032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0174e-68cb-4bf7-8f2d-20895225c08d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:21.957406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0174e-68cb-4bf7-8f2d-20895225c08d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:21.964973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bebab28-f8c4-4a4e-bc51-c39b59552dc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:21.965767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bebab28-f8c4-4a4e-bc51-c39b59552dc4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:21.966047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bebab28-f8c4-4a4e-bc51-c39b59552dc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:21.967612 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bebab28-f8c4-4a4e-bc51-c39b59552dc4 admin admin] PATH_INFO: `/v3/projects/00f3432914874e9699e2c7b9b7917e44/users/a22fcf5c814a452bace976751ceb10d7/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:21.972172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bebab28-f8c4-4a4e-bc51-c39b59552dc4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00f3432914874e9699e2c7b9b7917e44, user_id=a22fcf5c814a452bace976751ceb10d7, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:21.977032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bebab28-f8c4-4a4e-bc51-c39b59552dc4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:21.994002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 819/1626] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:21 2022] PUT /identity/v3/projects/00f3432914874e9699e2c7b9b7917e44/users/a22fcf5c814a452bace976751ceb10d7/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:13:22.008451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c732a-5b08-40ac-b6cc-ff9ac1890ed3 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:22.008890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c732a-5b08-40ac-b6cc-ff9ac1890ed3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:22.009405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c732a-5b08-40ac-b6cc-ff9ac1890ed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:22.035190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86754d4c-09ca-4b8c-9416-7694be232858 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:22.036609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86754d4c-09ca-4b8c-9416-7694be232858 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:22.036893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86754d4c-09ca-4b8c-9416-7694be232858 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:22.044839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 820/1627] 10.222.0.16 () {54 vars in 912 bytes} [Thu Aug 25 05:13:22 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 05:13:22.065353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cadded-5e86-43c6-9b3f-41dbe9fd663a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:22.066343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cadded-5e86-43c6-9b3f-41dbe9fd663a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:22.068311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cadded-5e86-43c6-9b3f-41dbe9fd663a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:22.068612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-9b3c732a-5b08-40ac-b6cc-ff9ac1890ed3 None None] MFA Rules not processed for user `a22fcf5c814a452bace976751ceb10d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:22.069729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9b3c732a-5b08-40ac-b6cc-ff9ac1890ed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:22.101169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:22.121105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-17cadded-5e86-43c6-9b3f-41dbe9fd663a None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:22.121105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-17cadded-5e86-43c6-9b3f-41dbe9fd663a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:22.235909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 808/1628] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:22 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:22.244091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c732a-5b08-40ac-b6cc-ff9ac1890ed3 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:22.255064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 821/1629] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:13:22 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:13:22.307190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c732a-5b08-40ac-b6cc-ff9ac1890ed3 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:22.313518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beca424-8bbc-428f-a63d-356fbc18d5e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:22.314311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beca424-8bbc-428f-a63d-356fbc18d5e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:22.314660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beca424-8bbc-428f-a63d-356fbc18d5e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:22.315010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beca424-8bbc-428f-a63d-356fbc18d5e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:22.331769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beca424-8bbc-428f-a63d-356fbc18d5e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:22.333528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beca424-8bbc-428f-a63d-356fbc18d5e7 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:22.340636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:22.348922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 809/1630] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:22 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:13:22.450769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cadded-5e86-43c6-9b3f-41dbe9fd663a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:22.460537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cadded-5e86-43c6-9b3f-41dbe9fd663a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:22.465778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b85a92-94fa-46b0-bca9-d4568f9e3d28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:22.466486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b85a92-94fa-46b0-bca9-d4568f9e3d28 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:22.466924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b85a92-94fa-46b0-bca9-d4568f9e3d28 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:22.467742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b85a92-94fa-46b0-bca9-d4568f9e3d28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:22.485148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b85a92-94fa-46b0-bca9-d4568f9e3d28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:22.485148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b85a92-94fa-46b0-bca9-d4568f9e3d28 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:22.496075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 822/1631] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:23.210061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beca424-8bbc-428f-a63d-356fbc18d5e7 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:23.210061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beca424-8bbc-428f-a63d-356fbc18d5e7 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:23.210061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:23.210061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:23.245324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:23.245324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] PATH_INFO: `/v3/users/f112c0816b414fca88217af02a6f3f9c` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:23.245324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f112c0816b414fca88217af02a6f3f9c)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:23.245324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:23.610996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f112c0816b414fca88217af02a6f3f9c'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:23.641131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f112c0816b414fca88217af02a6f3f9c'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:23.642116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f112c0816b414fca88217af02a6f3f9c'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:23.764766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f112c0816b414fca88217af02a6f3f9c'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:23.776983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f112c0816b414fca88217af02a6f3f9c'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:23.791148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 810/1632] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:23 2022] DELETE /identity/v3/users/f112c0816b414fca88217af02a6f3f9c => generated 0 bytes in 757 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:23.795815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35c07d0-6540-4164-b39e-87683e54188c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:23.796362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35c07d0-6540-4164-b39e-87683e54188c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:23.796362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35c07d0-6540-4164-b39e-87683e54188c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:23.816386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d35c07d0-6540-4164-b39e-87683e54188c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:23.817674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d35c07d0-6540-4164-b39e-87683e54188c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:24.024428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 823/1633] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:24.213577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:24.215906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:24.252402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9683fef-997f-419f-8f26-8db35fb4fbb7 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:24.253337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9683fef-997f-419f-8f26-8db35fb4fbb7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:24.253815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9683fef-997f-419f-8f26-8db35fb4fbb7 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:24.255406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 824/1634] 10.222.0.16 () {54 vars in 962 bytes} [Thu Aug 25 05:13:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 05:13:24.263960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f170b-bcbf-466c-8073-e6c720874d95 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:24.264217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f170b-bcbf-466c-8073-e6c720874d95 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:24.264498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f170b-bcbf-466c-8073-e6c720874d95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:24.294125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d50f170b-bcbf-466c-8073-e6c720874d95 None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:24.295417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d50f170b-bcbf-466c-8073-e6c720874d95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:24.343339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c7245a-3f26-47ad-8ec9-d91a06a70e01 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:24.353818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f6207f-886d-4303-9058-89388b4a9ada service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:24.354543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f6207f-886d-4303-9058-89388b4a9ada service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:24.356553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f6207f-886d-4303-9058-89388b4a9ada service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:24.356948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f6207f-886d-4303-9058-89388b4a9ada service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:24.387153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f6207f-886d-4303-9058-89388b4a9ada service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:24.395605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f6207f-886d-4303-9058-89388b4a9ada service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:24.425353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 811/1635] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:24.434397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 825/1636] 10.222.0.16 () {58 vars in 1059 bytes} [Thu Aug 25 05:13:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:24.442010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e130309b-a0cb-4d7f-bd41-40d7199ba24d None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:24.442157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e130309b-a0cb-4d7f-bd41-40d7199ba24d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:24.442415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e130309b-a0cb-4d7f-bd41-40d7199ba24d None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:24.452301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 812/1637] 10.222.0.16 () {54 vars in 912 bytes} [Thu Aug 25 05:13:24 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:13:24.455359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719872bb-6d02-4547-9e2e-3692e764794c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:24.455652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719872bb-6d02-4547-9e2e-3692e764794c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:24.455984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719872bb-6d02-4547-9e2e-3692e764794c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:24.489127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-719872bb-6d02-4547-9e2e-3692e764794c None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:24.490776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-719872bb-6d02-4547-9e2e-3692e764794c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:24.521905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 826/1638] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:13:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:24.530070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e130309b-a0cb-4d7f-bd41-40d7199ba24d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:24.537580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e130309b-a0cb-4d7f-bd41-40d7199ba24d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:24.555153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:24.667628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e130309b-a0cb-4d7f-bd41-40d7199ba24d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:24.692731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6380091d-b7e3-4635-a24b-f9a476df6524 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:24.693563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6380091d-b7e3-4635-a24b-f9a476df6524 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:24.694045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6380091d-b7e3-4635-a24b-f9a476df6524 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:24.694488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6380091d-b7e3-4635-a24b-f9a476df6524 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:24.727257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6380091d-b7e3-4635-a24b-f9a476df6524 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:24.728878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6380091d-b7e3-4635-a24b-f9a476df6524 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:24.751750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 813/1639] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:25.907514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719872bb-6d02-4547-9e2e-3692e764794c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:25.913000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-719872bb-6d02-4547-9e2e-3692e764794c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:25.940540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719872bb-6d02-4547-9e2e-3692e764794c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:25.956116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:25.963595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:25.963595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:25.963595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] PATH_INFO: `/v3/projects/4caa2607a9024662846dbce3fa4907db` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:25.963595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=4caa2607a9024662846dbce3fa4907db)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:25.973977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:25.976841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:26.206709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4caa2607a9024662846dbce3fa4907db', 'request_id': 'req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.210356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4caa2607a9024662846dbce3fa4907db', 'request_id': 'req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.296576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4caa2607a9024662846dbce3fa4907db', 'request_id': 'req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.318597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] The token cache is being invalidate because project 4caa2607a9024662846dbce3fa4907db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:26.318953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.319719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.325075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 827/1640] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:25 2022] DELETE /identity/v3/projects/4caa2607a9024662846dbce3fa4907db => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:26.332574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6380091d-b7e3-4635-a24b-f9a476df6524 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:26.346038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6380091d-b7e3-4635-a24b-f9a476df6524 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:26.588767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:26.589706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:26.590028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:26.590303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] PATH_INFO: `/v3/users/a22fcf5c814a452bace976751ceb10d7` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:26.595725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a22fcf5c814a452bace976751ceb10d7)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:26.598574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:26.853856 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a22fcf5c814a452bace976751ceb10d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.876393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a22fcf5c814a452bace976751ceb10d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.877394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a22fcf5c814a452bace976751ceb10d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.976275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a22fcf5c814a452bace976751ceb10d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:26.987668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a22fcf5c814a452bace976751ceb10d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:27.004482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 814/1641] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:26 2022] DELETE /identity/v3/users/a22fcf5c814a452bace976751ceb10d7 => generated 0 bytes in 676 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:27.686435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:27.688514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:27.720332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6c6e5c-ac04-497c-b4b9-a904d55331fd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:27.838156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:27.841385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:27.842231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:27.842771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] PATH_INFO: `/v3/users/1f0948bf60d44b8d8c26738d6339b058` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:27.846700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f0948bf60d44b8d8c26738d6339b058)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:27.848199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: WARNING py.warnings [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:13:27.848199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: warnings.warn(msg) Aug 25 05:13:27.848199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: Aug 25 05:13:27.850564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:27.981685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:27.983966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:28.048089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7498677c-af01-419a-867b-7b7cc56867e8 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:28.072442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:28.073157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:28.073433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:28.073939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] PATH_INFO: `/v3/projects/00f3432914874e9699e2c7b9b7917e44` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:28.077927 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] RBAC: Authorizing `identity:delete_project(project_id=00f3432914874e9699e2c7b9b7917e44)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:28.079471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: WARNING py.warnings [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 25 05:13:28.079471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: warnings.warn(msg) Aug 25 05:13:28.079471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: Aug 25 05:13:28.084977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:28.128448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f0948bf60d44b8d8c26738d6339b058'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.129304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f0948bf60d44b8d8c26738d6339b058'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.153511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:28.191006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f0948bf60d44b8d8c26738d6339b058'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.295301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f0948bf60d44b8d8c26738d6339b058'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.313047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00f3432914874e9699e2c7b9b7917e44', 'request_id': 'req-1a4e2c72-5c57-401e-a915-8cb17c21a777'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.330203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a27f3071-7e8e-4e39-9e10-27aa19155b6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f0948bf60d44b8d8c26738d6339b058'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.350398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 828/1642] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:27 2022] DELETE /identity/v3/users/1f0948bf60d44b8d8c26738d6339b058 => generated 0 bytes in 673 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:28.355807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d0ac4-6cc8-45ec-9cc9-ff01e6f16408 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:28.362923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d0ac4-6cc8-45ec-9cc9-ff01e6f16408 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:28.363287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12d0ac4-6cc8-45ec-9cc9-ff01e6f16408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:28.384465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00f3432914874e9699e2c7b9b7917e44', 'request_id': 'req-1a4e2c72-5c57-401e-a915-8cb17c21a777'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.409488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00f3432914874e9699e2c7b9b7917e44', 'request_id': 'req-1a4e2c72-5c57-401e-a915-8cb17c21a777'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.446976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f12d0ac4-6cc8-45ec-9cc9-ff01e6f16408 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:28.473777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f12d0ac4-6cc8-45ec-9cc9-ff01e6f16408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:28.479616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] The token cache is being invalidate because project 00f3432914874e9699e2c7b9b7917e44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:28.487389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.487389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a4e2c72-5c57-401e-a915-8cb17c21a777 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:28.493855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 815/1643] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:27 2022] DELETE /identity/v3/projects/00f3432914874e9699e2c7b9b7917e44 => generated 0 bytes in 513 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:13:28.494611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:28.506936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6804f-f158-4ce2-9932-81f657e97e7d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:28.507302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6804f-f158-4ce2-9932-81f657e97e7d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:28.507589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a6804f-f158-4ce2-9932-81f657e97e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:28.594958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c3a6804f-f158-4ce2-9932-81f657e97e7d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:28.596502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c3a6804f-f158-4ce2-9932-81f657e97e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:28.676707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 829/1644] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:28.763949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 816/1645] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:28.773303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05342504-9a1c-42d9-acfb-f426e9663324 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:28.773890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05342504-9a1c-42d9-acfb-f426e9663324 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:28.774039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05342504-9a1c-42d9-acfb-f426e9663324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:28.792387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a6804f-f158-4ce2-9932-81f657e97e7d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:28.792387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c3a6804f-f158-4ce2-9932-81f657e97e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:28.815444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-05342504-9a1c-42d9-acfb-f426e9663324 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:28.817058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-05342504-9a1c-42d9-acfb-f426e9663324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:28.846350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 830/1646] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:28.850143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05342504-9a1c-42d9-acfb-f426e9663324 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:28.881107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05342504-9a1c-42d9-acfb-f426e9663324 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:28.888369 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:28.890466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:28.890996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:28.891340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:28.892188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:28.892579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:28.896583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:28.945880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 831/1647] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:28.947266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a6804f-f158-4ce2-9932-81f657e97e7d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:28.952963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:28.966395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5be33-988f-4743-8113-7240a1f72696 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:28.967635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5be33-988f-4743-8113-7240a1f72696 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:28.967635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5be33-988f-4743-8113-7240a1f72696 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:28.967635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5be33-988f-4743-8113-7240a1f72696 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:28.971662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1929fd8c-11e9-417b-86cb-86ea8e003a78 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:28.991257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae10e3f-df2d-4990-8d01-f9266755627c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:28.992115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae10e3f-df2d-4990-8d01-f9266755627c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:28.992565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae10e3f-df2d-4990-8d01-f9266755627c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:28.993146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae10e3f-df2d-4990-8d01-f9266755627c admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:28.994086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae10e3f-df2d-4990-8d01-f9266755627c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:28.994976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d5be33-988f-4743-8113-7240a1f72696 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:28.997773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae10e3f-df2d-4990-8d01-f9266755627c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:29.002646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d5be33-988f-4743-8113-7240a1f72696 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:29.031819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 817/1648] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:29.115808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 832/1649] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:28 2022] POST /identity/v3/projects => generated 365 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:29.120549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5be33-988f-4743-8113-7240a1f72696 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:29.167711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5be33-988f-4743-8113-7240a1f72696 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:29.444491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:29.475826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3f8137-e21b-4275-a60c-ef4f14e4eadd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:29.476473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f8137-e21b-4275-a60c-ef4f14e4eadd admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:29.483707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f8137-e21b-4275-a60c-ef4f14e4eadd admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:29.483707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f8137-e21b-4275-a60c-ef4f14e4eadd admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:29.483707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3f8137-e21b-4275-a60c-ef4f14e4eadd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:29.490129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3f8137-e21b-4275-a60c-ef4f14e4eadd admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:29.640756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 818/1650] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:29 2022] POST /identity/v3/users => generated 283 bytes in 521 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:29.641757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae10e3f-df2d-4990-8d01-f9266755627c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:29.672655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae10e3f-df2d-4990-8d01-f9266755627c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:29.701319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8113aa2e-7f70-446d-9a84-066763a5ba19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:29.702005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113aa2e-7f70-446d-9a84-066763a5ba19 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:29.702321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113aa2e-7f70-446d-9a84-066763a5ba19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:29.703281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113aa2e-7f70-446d-9a84-066763a5ba19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:29.703985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8113aa2e-7f70-446d-9a84-066763a5ba19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:29.705687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8113aa2e-7f70-446d-9a84-066763a5ba19 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:29.746218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 833/1651] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:29 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:29.747509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3f8137-e21b-4275-a60c-ef4f14e4eadd admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:29.783986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3f8137-e21b-4275-a60c-ef4f14e4eadd admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:29.802675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562c983f-4cb4-4ff1-b0d9-b9cdd167395a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:29.811122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c983f-4cb4-4ff1-b0d9-b9cdd167395a admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:29.811529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c983f-4cb4-4ff1-b0d9-b9cdd167395a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:29.812823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562c983f-4cb4-4ff1-b0d9-b9cdd167395a admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:29.814862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562c983f-4cb4-4ff1-b0d9-b9cdd167395a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:29.821608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562c983f-4cb4-4ff1-b0d9-b9cdd167395a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:29.865695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8113aa2e-7f70-446d-9a84-066763a5ba19 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:29.866251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 819/1652] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:29 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:13:29.895728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8113aa2e-7f70-446d-9a84-066763a5ba19 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:29.919932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a41a381-1710-4c9b-afb6-d1ef1f63c4d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:29.922023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41a381-1710-4c9b-afb6-d1ef1f63c4d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:29.922307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41a381-1710-4c9b-afb6-d1ef1f63c4d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:29.922676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41a381-1710-4c9b-afb6-d1ef1f63c4d1 admin admin] PATH_INFO: `/v3/projects/c212520a3f1145f8a89af8f197d83b76/users/0fba0a088efa4acebc0475e44fa0f7be/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:29.959524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a41a381-1710-4c9b-afb6-d1ef1f63c4d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c212520a3f1145f8a89af8f197d83b76, user_id=0fba0a088efa4acebc0475e44fa0f7be, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:29.963515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a41a381-1710-4c9b-afb6-d1ef1f63c4d1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:30.078909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ba802-4eb7-4e7e-8dba-e68102e7748a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:30.079902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ba802-4eb7-4e7e-8dba-e68102e7748a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:30.080260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 834/1653] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:29 2022] PUT /identity/v3/projects/c212520a3f1145f8a89af8f197d83b76/users/0fba0a088efa4acebc0475e44fa0f7be/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:30.080475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ba802-4eb7-4e7e-8dba-e68102e7748a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:30.151508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-0f9ba802-4eb7-4e7e-8dba-e68102e7748a None None] MFA Rules not processed for user `0fba0a088efa4acebc0475e44fa0f7be`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:30.159668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0f9ba802-4eb7-4e7e-8dba-e68102e7748a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:30.239717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fabf161-f088-41e6-894f-a05cb6913ff9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:30.240047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fabf161-f088-41e6-894f-a05cb6913ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:30.240325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fabf161-f088-41e6-894f-a05cb6913ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:30.290644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 820/1654] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:30 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:30.295581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9ba802-4eb7-4e7e-8dba-e68102e7748a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:30.303307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8fabf161-f088-41e6-894f-a05cb6913ff9 None None] MFA Rules not processed for user `3107aef7eb954ac4845c8903b005ce93`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:30.304859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8fabf161-f088-41e6-894f-a05cb6913ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:30.470388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 835/1655] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:30.481495 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b77e3ef-e874-48c0-9d4b-4f86857c7c7c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:30.484127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b77e3ef-e874-48c0-9d4b-4f86857c7c7c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:30.484657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b77e3ef-e874-48c0-9d4b-4f86857c7c7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:30.519679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9ba802-4eb7-4e7e-8dba-e68102e7748a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:30.549612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df25c66d-2800-4a6b-89da-92a8ea786386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:30.553792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df25c66d-2800-4a6b-89da-92a8ea786386 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:30.556686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df25c66d-2800-4a6b-89da-92a8ea786386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:30.558044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df25c66d-2800-4a6b-89da-92a8ea786386 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:30.559008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df25c66d-2800-4a6b-89da-92a8ea786386 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:30.560507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8b77e3ef-e874-48c0-9d4b-4f86857c7c7c None None] MFA Rules not processed for user `3107aef7eb954ac4845c8903b005ce93`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:30.564141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8b77e3ef-e874-48c0-9d4b-4f86857c7c7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:30.568176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df25c66d-2800-4a6b-89da-92a8ea786386 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:30.589516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 836/1656] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:30.607470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b77e3ef-e874-48c0-9d4b-4f86857c7c7c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:30.609042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8b77e3ef-e874-48c0-9d4b-4f86857c7c7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:30.650571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:30.709054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 821/1657] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:30 2022] POST /identity/v3/projects => generated 365 bytes in 414 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:30.710252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df25c66d-2800-4a6b-89da-92a8ea786386 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:30.793149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b77e3ef-e874-48c0-9d4b-4f86857c7c7c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:30.883597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df25c66d-2800-4a6b-89da-92a8ea786386 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:30.888525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa5f057-7d33-4d10-9eed-89f4dac18717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:30.889209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa5f057-7d33-4d10-9eed-89f4dac18717 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:30.889491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa5f057-7d33-4d10-9eed-89f4dac18717 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:30.890525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa5f057-7d33-4d10-9eed-89f4dac18717 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:30.891171 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa5f057-7d33-4d10-9eed-89f4dac18717 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:30.893131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa5f057-7d33-4d10-9eed-89f4dac18717 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:30.918107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8a74f9-8ec1-4185-8fee-b9ab690ef219 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:30.919039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a74f9-8ec1-4185-8fee-b9ab690ef219 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:30.919774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a74f9-8ec1-4185-8fee-b9ab690ef219 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:30.920169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8a74f9-8ec1-4185-8fee-b9ab690ef219 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:31.023994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 822/1658] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:30 2022] POST /identity/v3/users => generated 289 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:31.024960 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa5f057-7d33-4d10-9eed-89f4dac18717 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:31.052115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa5f057-7d33-4d10-9eed-89f4dac18717 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:31.058318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234b5d43-dbac-4706-950b-afdd07f08818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:31.059256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234b5d43-dbac-4706-950b-afdd07f08818 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:31.059539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234b5d43-dbac-4706-950b-afdd07f08818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:31.060393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234b5d43-dbac-4706-950b-afdd07f08818 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:31.061060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234b5d43-dbac-4706-950b-afdd07f08818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:31.062341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8a74f9-8ec1-4185-8fee-b9ab690ef219 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:31.066661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234b5d43-dbac-4706-950b-afdd07f08818 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:31.069466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8a74f9-8ec1-4185-8fee-b9ab690ef219 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:31.107140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 837/1659] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:30 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:13:31.109876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 823/1660] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:31 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:31.118660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8a74f9-8ec1-4185-8fee-b9ab690ef219 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:31.125502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:31.149947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8a74f9-8ec1-4185-8fee-b9ab690ef219 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:31.196202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:31.197348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:31.205904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:31.206261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] PATH_INFO: `/v3/projects/32057f9a02074fdabcf858de7e733bc0/users/df764242ae374ef1b119974d4dcbe0b1/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:31.231163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32057f9a02074fdabcf858de7e733bc0, user_id=df764242ae374ef1b119974d4dcbe0b1, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:31.238460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:31.362551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 838/1661] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:31 2022] PUT /identity/v3/projects/32057f9a02074fdabcf858de7e733bc0/users/df764242ae374ef1b119974d4dcbe0b1/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:31.366661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234b5d43-dbac-4706-950b-afdd07f08818 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:31.379872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234b5d43-dbac-4706-950b-afdd07f08818 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:31.381145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:31.384221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:31.527572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11290e-3af2-4e12-a5b4-ff27aa9e1a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:31.529639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11290e-3af2-4e12-a5b4-ff27aa9e1a04 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:31.529925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11290e-3af2-4e12-a5b4-ff27aa9e1a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:31.530201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e11290e-3af2-4e12-a5b4-ff27aa9e1a04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:31.531396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11290e-3af2-4e12-a5b4-ff27aa9e1a04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:31.533028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e11290e-3af2-4e12-a5b4-ff27aa9e1a04 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:31.548757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea48d7f-6567-4163-a375-604f206a1730 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:31.560728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 824/1662] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:31 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:31.561737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11290e-3af2-4e12-a5b4-ff27aa9e1a04 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:31.564194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:31.564962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:31.565300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:31.565502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] PATH_INFO: `/v3/projects/4b9739a947184e05b94cfd3b305bd9f5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:31.571559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b9739a947184e05b94cfd3b305bd9f5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:31.573554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:31.584610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e11290e-3af2-4e12-a5b4-ff27aa9e1a04 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:31.598908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcd83f1-20d2-4146-885f-98eb16224f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:31.599658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd83f1-20d2-4146-885f-98eb16224f2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:31.599940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd83f1-20d2-4146-885f-98eb16224f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:31.600215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcd83f1-20d2-4146-885f-98eb16224f2d admin admin] PATH_INFO: `/v3/projects/32057f9a02074fdabcf858de7e733bc0/users/df764242ae374ef1b119974d4dcbe0b1/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:31.603096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcd83f1-20d2-4146-885f-98eb16224f2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=32057f9a02074fdabcf858de7e733bc0, user_id=df764242ae374ef1b119974d4dcbe0b1, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:31.609503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcd83f1-20d2-4146-885f-98eb16224f2d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:31.642582 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 825/1663] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:31 2022] PUT /identity/v3/projects/32057f9a02074fdabcf858de7e733bc0/users/df764242ae374ef1b119974d4dcbe0b1/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:31.657635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ec6fbe-9208-4549-9322-555d7e3f67e1 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:31.657635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ec6fbe-9208-4549-9322-555d7e3f67e1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:31.657635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ec6fbe-9208-4549-9322-555d7e3f67e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:31.704684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-35ec6fbe-9208-4549-9322-555d7e3f67e1 None None] MFA Rules not processed for user `df764242ae374ef1b119974d4dcbe0b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:31.706614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-35ec6fbe-9208-4549-9322-555d7e3f67e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:31.759453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b9739a947184e05b94cfd3b305bd9f5', 'request_id': 'req-5cecb290-3635-4791-ae53-641c43b432bb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:31.774450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b9739a947184e05b94cfd3b305bd9f5', 'request_id': 'req-5cecb290-3635-4791-ae53-641c43b432bb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:31.835271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:31.856390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 826/1664] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:31 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:31.869441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ec6fbe-9208-4549-9322-555d7e3f67e1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:31.871202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-35ec6fbe-9208-4549-9322-555d7e3f67e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:31.873398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b9739a947184e05b94cfd3b305bd9f5', 'request_id': 'req-5cecb290-3635-4791-ae53-641c43b432bb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:31.903001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] The token cache is being invalidate because project 4b9739a947184e05b94cfd3b305bd9f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:31.903527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:31.906621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5cecb290-3635-4791-ae53-641c43b432bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:31.920531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 839/1665] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:31 2022] DELETE /identity/v3/projects/4b9739a947184e05b94cfd3b305bd9f5 => generated 0 bytes in 548 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:31.923840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2592c17-0b1c-4857-86c1-4a9b2aad00a5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:31.924133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2592c17-0b1c-4857-86c1-4a9b2aad00a5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:31.924414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2592c17-0b1c-4857-86c1-4a9b2aad00a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:31.956528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ec6fbe-9208-4549-9322-555d7e3f67e1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:31.973739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f2592c17-0b1c-4857-86c1-4a9b2aad00a5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:31.975065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f2592c17-0b1c-4857-86c1-4a9b2aad00a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:32.088088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 840/1666] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:32.096303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d029a-d463-4054-b494-cdf6d463104d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.101819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d029a-d463-4054-b494-cdf6d463104d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.102319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095d029a-d463-4054-b494-cdf6d463104d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.105184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3365828-daad-4dba-b261-0e0aec24b62f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.107292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3365828-daad-4dba-b261-0e0aec24b62f service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.107594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3365828-daad-4dba-b261-0e0aec24b62f service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.109075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3365828-daad-4dba-b261-0e0aec24b62f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.114029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b3365828-daad-4dba-b261-0e0aec24b62f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:32.141843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-095d029a-d463-4054-b494-cdf6d463104d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:32.143097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-095d029a-d463-4054-b494-cdf6d463104d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:32.183688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 841/1667] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:13:32.185173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c16907-6fa5-42d5-9893-04859de70f89 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.185496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c16907-6fa5-42d5-9893-04859de70f89 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.185794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c16907-6fa5-42d5-9893-04859de70f89 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.187820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 842/1668] 10.222.0.16 () {54 vars in 912 bytes} [Thu Aug 25 05:13:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:13:32.189859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c16907-6fa5-42d5-9893-04859de70f89 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:32.223251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3365828-daad-4dba-b261-0e0aec24b62f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:32.228386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c16907-6fa5-42d5-9893-04859de70f89 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:32.231897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3365828-daad-4dba-b261-0e0aec24b62f service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:32.232131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.232749 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.233090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.233444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.234184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:32.234488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:32.236129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:32.244850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 827/1669] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:31 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:13:32.246837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0f5c31-e716-4cf6-ad22-b715ddc370e4 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.247163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0f5c31-e716-4cf6-ad22-b715ddc370e4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.247462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0f5c31-e716-4cf6-ad22-b715ddc370e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.288190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 843/1670] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:32.292211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:32.305072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5c0f5c31-e716-4cf6-ad22-b715ddc370e4 None None] MFA Rules not processed for user `9220ba3637a542a382ad1d42ff1ff459`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:32.305285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25ed4bb-392a-4305-b1a0-f6e26c2633ae admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:32.306517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5c0f5c31-e716-4cf6-ad22-b715ddc370e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:32.311508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f92e84-b811-4e6e-b9ac-8e73f2447e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.313963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f92e84-b811-4e6e-b9ac-8e73f2447e03 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.313963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f92e84-b811-4e6e-b9ac-8e73f2447e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.313963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f92e84-b811-4e6e-b9ac-8e73f2447e03 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.314234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:32.314499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f92e84-b811-4e6e-b9ac-8e73f2447e03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:32.317544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f92e84-b811-4e6e-b9ac-8e73f2447e03 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:32.406830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 828/1671] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:13:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:32.430384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0f5c31-e716-4cf6-ad22-b715ddc370e4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:32.432768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5c0f5c31-e716-4cf6-ad22-b715ddc370e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:32.437617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 844/1672] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:32 2022] POST /identity/v3/projects => generated 385 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:13:32.441299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f92e84-b811-4e6e-b9ac-8e73f2447e03 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:32.455072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0f5c31-e716-4cf6-ad22-b715ddc370e4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:32.549917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.551645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f92e84-b811-4e6e-b9ac-8e73f2447e03 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:32.552153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.552580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.552993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.564196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfd1a9f-24c0-4a30-b78b-c3678ba094c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.564932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfd1a9f-24c0-4a30-b78b-c3678ba094c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.566616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfd1a9f-24c0-4a30-b78b-c3678ba094c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.567256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfd1a9f-24c0-4a30-b78b-c3678ba094c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.567375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:32.568192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfd1a9f-24c0-4a30-b78b-c3678ba094c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:32.569108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:32.572497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfd1a9f-24c0-4a30-b78b-c3678ba094c3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:32.582977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 829/1673] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:32.583945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:32.589505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:32.684896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431f18ef-1aee-4f6d-a3f9-0e61726dd49c service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:32.686929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.687115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.687396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.687672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.690806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:32.780906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:32.785096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:32.796418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 830/1674] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:32 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:32.810868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 845/1675] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:32 2022] POST /identity/v3/users => generated 293 bytes in 366 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:13:32.813062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:32.839707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72285eaa-e758-4bb3-a981-ac1b0ee96da3 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:32.853108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b436beac-c6a9-4a32-a8fa-b727de720f2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.853807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b436beac-c6a9-4a32-a8fa-b727de720f2a admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.854085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b436beac-c6a9-4a32-a8fa-b727de720f2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.854358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b436beac-c6a9-4a32-a8fa-b727de720f2a admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.855004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b436beac-c6a9-4a32-a8fa-b727de720f2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:32.860168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b436beac-c6a9-4a32-a8fa-b727de720f2a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:32.889052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 831/1676] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:32 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:32.892550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfd1a9f-24c0-4a30-b78b-c3678ba094c3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:32.909061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfd1a9f-24c0-4a30-b78b-c3678ba094c3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:32.925442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626f7f3e-8ce5-4253-b0de-084027b6d76f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.926867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f7f3e-8ce5-4253-b0de-084027b6d76f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.927501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f7f3e-8ce5-4253-b0de-084027b6d76f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.927990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626f7f3e-8ce5-4253-b0de-084027b6d76f admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:32.928946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626f7f3e-8ce5-4253-b0de-084027b6d76f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:32.931559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626f7f3e-8ce5-4253-b0de-084027b6d76f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:32.959861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 846/1677] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:32 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:32.965028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b436beac-c6a9-4a32-a8fa-b727de720f2a admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:32.979493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b436beac-c6a9-4a32-a8fa-b727de720f2a admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:32.993240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d8e79f-d02b-4e5e-8aa3-61f06a1349f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:32.994189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8e79f-d02b-4e5e-8aa3-61f06a1349f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:32.996609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8e79f-d02b-4e5e-8aa3-61f06a1349f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:32.997097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d8e79f-d02b-4e5e-8aa3-61f06a1349f1 admin admin] PATH_INFO: `/v3/projects/ac2508877957498687af6f2e55b7b585/users/b7940a92d02844559cc1ebd3c5bcaf38/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:33.049978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d8e79f-d02b-4e5e-8aa3-61f06a1349f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac2508877957498687af6f2e55b7b585, user_id=b7940a92d02844559cc1ebd3c5bcaf38, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:33.056912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d8e79f-d02b-4e5e-8aa3-61f06a1349f1 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:33.148988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 832/1678] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:32 2022] PUT /identity/v3/projects/ac2508877957498687af6f2e55b7b585/users/b7940a92d02844559cc1ebd3c5bcaf38/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:33.159634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac5c28-ed86-4828-91b2-8a37f79a8f0c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:33.159924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac5c28-ed86-4828-91b2-8a37f79a8f0c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:33.160256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac5c28-ed86-4828-91b2-8a37f79a8f0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:33.192342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:33.214562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-fcac5c28-ed86-4828-91b2-8a37f79a8f0c None None] MFA Rules not processed for user `b7940a92d02844559cc1ebd3c5bcaf38`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:33.215790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fcac5c28-ed86-4828-91b2-8a37f79a8f0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:33.387383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 847/1679] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:33.398548 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d8e79f-d02b-4e5e-8aa3-61f06a1349f1 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:33.554601 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:33.615720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c025f5-7d7b-450e-a8b6-202f371bff3a None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:33.616243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c025f5-7d7b-450e-a8b6-202f371bff3a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:33.617403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c025f5-7d7b-450e-a8b6-202f371bff3a None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:33.619355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 848/1680] 10.222.0.16 () {54 vars in 961 bytes} [Thu Aug 25 05:13:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 25 05:13:33.630364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c9e1e-830d-4e89-87ab-3c6c51161f36 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:33.630364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c9e1e-830d-4e89-87ab-3c6c51161f36 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:33.630703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c9e1e-830d-4e89-87ab-3c6c51161f36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:33.648447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d8e79f-d02b-4e5e-8aa3-61f06a1349f1 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:33.665498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fdcd1f-f415-46ac-be3c-e88a73d13430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:33.681422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fdcd1f-f415-46ac-be3c-e88a73d13430 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:33.681422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fdcd1f-f415-46ac-be3c-e88a73d13430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:33.681422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fdcd1f-f415-46ac-be3c-e88a73d13430 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:33.681422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fdcd1f-f415-46ac-be3c-e88a73d13430 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:33.681422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fdcd1f-f415-46ac-be3c-e88a73d13430 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:33.706685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e96c9e1e-830d-4e89-87ab-3c6c51161f36 None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:33.710038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e96c9e1e-830d-4e89-87ab-3c6c51161f36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:33.828959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 833/1681] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:33 2022] POST /identity/v3/projects => generated 385 bytes in 438 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:33.838894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fdcd1f-f415-46ac-be3c-e88a73d13430 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:33.841624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 849/1682] 10.222.0.16 () {58 vars in 1058 bytes} [Thu Aug 25 05:13:33 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Aug 25 05:13:33.843739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96c9e1e-830d-4e89-87ab-3c6c51161f36 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:33.871326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96c9e1e-830d-4e89-87ab-3c6c51161f36 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:33.969662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fdcd1f-f415-46ac-be3c-e88a73d13430 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:33.974830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09ed26-32ee-49e4-a908-1e06bb713588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:33.975747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09ed26-32ee-49e4-a908-1e06bb713588 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:33.976027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09ed26-32ee-49e4-a908-1e06bb713588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:33.976301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09ed26-32ee-49e4-a908-1e06bb713588 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:33.977036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f09ed26-32ee-49e4-a908-1e06bb713588 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:33.979162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f09ed26-32ee-49e4-a908-1e06bb713588 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:34.017212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:34.017898 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:34.018260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:34.018649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:34.020896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:34.047517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:34.049629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:34.067339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 834/1683] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:33 2022] POST /identity/v3/users => generated 299 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:34.069195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09ed26-32ee-49e4-a908-1e06bb713588 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:34.074667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 850/1684] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:34.100963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09ed26-32ee-49e4-a908-1e06bb713588 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:34.116350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b81e5cb-9ebe-4657-a0ca-6d32db614c62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:34.117002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b81e5cb-9ebe-4657-a0ca-6d32db614c62 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:34.117288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b81e5cb-9ebe-4657-a0ca-6d32db614c62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:34.117564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b81e5cb-9ebe-4657-a0ca-6d32db614c62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:34.118243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b81e5cb-9ebe-4657-a0ca-6d32db614c62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:34.120304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b81e5cb-9ebe-4657-a0ca-6d32db614c62 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:34.151382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 835/1685] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:13:34.152801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:34.170436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:34.171497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b8fead-5834-4761-84a5-56ec8238edd7 service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:34.189390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74d14bd-d452-4006-b9d6-753f8e60ca6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:34.193091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d14bd-d452-4006-b9d6-753f8e60ca6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:34.193391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d14bd-d452-4006-b9d6-753f8e60ca6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:34.193760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d14bd-d452-4006-b9d6-753f8e60ca6d admin admin] PATH_INFO: `/v3/projects/2683783126e649808c64d06244f531ac/users/a5bf61fe6763450890369a56e9425e39/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:34.257107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74d14bd-d452-4006-b9d6-753f8e60ca6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2683783126e649808c64d06244f531ac, user_id=a5bf61fe6763450890369a56e9425e39, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:34.259866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74d14bd-d452-4006-b9d6-753f8e60ca6d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:34.414733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 851/1686] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:34 2022] PUT /identity/v3/projects/2683783126e649808c64d06244f531ac/users/a5bf61fe6763450890369a56e9425e39/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:34.416175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b81e5cb-9ebe-4657-a0ca-6d32db614c62 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:34.435665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b81e5cb-9ebe-4657-a0ca-6d32db614c62 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:34.656499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bac9762-ab31-4d8a-94db-695865248ef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:34.656961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac9762-ab31-4d8a-94db-695865248ef8 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:34.657290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac9762-ab31-4d8a-94db-695865248ef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:34.657572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac9762-ab31-4d8a-94db-695865248ef8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:34.658260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bac9762-ab31-4d8a-94db-695865248ef8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:34.660423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bac9762-ab31-4d8a-94db-695865248ef8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:34.683359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 836/1687] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:34.688517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74d14bd-d452-4006-b9d6-753f8e60ca6d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:34.703893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74d14bd-d452-4006-b9d6-753f8e60ca6d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:34.718596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2266b6b-1a5c-4c2d-8243-7eca5a34b573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:34.719508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2266b6b-1a5c-4c2d-8243-7eca5a34b573 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:34.719963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2266b6b-1a5c-4c2d-8243-7eca5a34b573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:34.720554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2266b6b-1a5c-4c2d-8243-7eca5a34b573 admin admin] PATH_INFO: `/v3/projects/2683783126e649808c64d06244f531ac/users/a5bf61fe6763450890369a56e9425e39/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:34.725813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2266b6b-1a5c-4c2d-8243-7eca5a34b573 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2683783126e649808c64d06244f531ac, user_id=a5bf61fe6763450890369a56e9425e39, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:34.731525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2266b6b-1a5c-4c2d-8243-7eca5a34b573 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:34.759541 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 852/1688] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:34 2022] PUT /identity/v3/projects/2683783126e649808c64d06244f531ac/users/a5bf61fe6763450890369a56e9425e39/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:13:34.771720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866bffeb-359c-4895-b2f7-8839165fc558 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:34.772009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866bffeb-359c-4895-b2f7-8839165fc558 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:34.772292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866bffeb-359c-4895-b2f7-8839165fc558 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:34.801536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:34.865166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-866bffeb-359c-4895-b2f7-8839165fc558 None None] MFA Rules not processed for user `a5bf61fe6763450890369a56e9425e39`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:34.866651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-866bffeb-359c-4895-b2f7-8839165fc558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:35.113940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 837/1689] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:34 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:35.126140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2266b6b-1a5c-4c2d-8243-7eca5a34b573 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:35.294758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:35.296475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2266b6b-1a5c-4c2d-8243-7eca5a34b573 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:35.317531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:35.318402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:35.318803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:35.322815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:35.352489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:35.356648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:35.391010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 853/1690] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:35 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:36.186802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6ac8-8466-40b6-be6d-b3b5dfb1bf57 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:36.187145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6ac8-8466-40b6-be6d-b3b5dfb1bf57 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:36.187428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6ac8-8466-40b6-be6d-b3b5dfb1bf57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:36.275303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-243f6ac8-8466-40b6-be6d-b3b5dfb1bf57 None None] MFA Rules not processed for user `ec39d995899c4d43bb4f8ffd9e6d2ef3`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:36.276549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-243f6ac8-8466-40b6-be6d-b3b5dfb1bf57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:36.447029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 838/1691] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:13:36.468459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:36.471016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:36.657257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd6377a-0432-4d81-973f-ac3135418785 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:36.676592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:36.677855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:36.679948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:36.687548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:36.703322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:36.705764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:36.714281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:36.732714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 854/1692] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:38.129111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243f6ac8-8466-40b6-be6d-b3b5dfb1bf57 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:38.395643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243f6ac8-8466-40b6-be6d-b3b5dfb1bf57 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:38.410028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:38.414713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:38.415976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:38.417625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] PATH_INFO: `/v3/users/b7940a92d02844559cc1ebd3c5bcaf38` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:38.422819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7940a92d02844559cc1ebd3c5bcaf38)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:38.436165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:38.796079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:38.797715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:38.823510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4935e9-555f-4ba1-ad3c-de16538dc944 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:38.836816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:38.837643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:38.837932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:38.838207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] PATH_INFO: `/v3/users/0fba0a088efa4acebc0475e44fa0f7be` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:38.844817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fba0a088efa4acebc0475e44fa0f7be)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:38.847301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:38.956228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7940a92d02844559cc1ebd3c5bcaf38'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:38.974032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7940a92d02844559cc1ebd3c5bcaf38'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:38.975364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7940a92d02844559cc1ebd3c5bcaf38'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.114150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7940a92d02844559cc1ebd3c5bcaf38'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.117285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:39.127741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-eff2b207-3c4b-4978-967b-a899bb231fa3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7940a92d02844559cc1ebd3c5bcaf38'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.146687 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 839/1693] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:38 2022] DELETE /identity/v3/users/b7940a92d02844559cc1ebd3c5bcaf38 => generated 0 bytes in 1018 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:39.152722 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bf8601-2257-4a46-852f-18586dd053bf None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:39.153170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bf8601-2257-4a46-852f-18586dd053bf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:39.153696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bf8601-2257-4a46-852f-18586dd053bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:39.179440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-73bf8601-2257-4a46-852f-18586dd053bf None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:39.183304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-73bf8601-2257-4a46-852f-18586dd053bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:39.280184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fba0a088efa4acebc0475e44fa0f7be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.281056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fba0a088efa4acebc0475e44fa0f7be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.288592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 840/1694] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:39.318875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bf8601-2257-4a46-852f-18586dd053bf None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:39.321441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-73bf8601-2257-4a46-852f-18586dd053bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:39.333110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fba0a088efa4acebc0475e44fa0f7be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.348168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bf8601-2257-4a46-852f-18586dd053bf None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:39.364186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:39.365045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:39.365398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:39.365780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] PATH_INFO: `/v3/users/bc62fa0058a340f882759c14873a0be5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:39.368087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc62fa0058a340f882759c14873a0be5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:39.375364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:39.421615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fba0a088efa4acebc0475e44fa0f7be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.434562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fba0a088efa4acebc0475e44fa0f7be'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.446288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 855/1695] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:38 2022] DELETE /identity/v3/users/0fba0a088efa4acebc0475e44fa0f7be => generated 0 bytes in 651 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:39.447164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:39.448523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:39.603022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1521e1-5c8f-4c04-8d8c-23fbcb2941e8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:39.614034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8222548b-5555-4ef9-be4c-f5b9e97e0add service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:39.617203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8222548b-5555-4ef9-be4c-f5b9e97e0add service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:39.617943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8222548b-5555-4ef9-be4c-f5b9e97e0add service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:39.618332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8222548b-5555-4ef9-be4c-f5b9e97e0add service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:39.748458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc62fa0058a340f882759c14873a0be5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.790095 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc62fa0058a340f882759c14873a0be5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.791248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc62fa0058a340f882759c14873a0be5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.796837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8222548b-5555-4ef9-be4c-f5b9e97e0add service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:39.798441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8222548b-5555-4ef9-be4c-f5b9e97e0add service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:39.813036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 856/1696] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:39.814702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4113f-5318-4b2f-93c1-f0c04521667b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:39.815159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4113f-5318-4b2f-93c1-f0c04521667b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:39.815571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d4113f-5318-4b2f-93c1-f0c04521667b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:39.845251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a0d4113f-5318-4b2f-93c1-f0c04521667b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:39.847196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a0d4113f-5318-4b2f-93c1-f0c04521667b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:39.872183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 857/1697] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:39.891423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc62fa0058a340f882759c14873a0be5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.912741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7896494e-3c97-4cfb-8f43-3eaadb7c48f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc62fa0058a340f882759c14873a0be5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:39.963929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 841/1698] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:39 2022] DELETE /identity/v3/users/bc62fa0058a340f882759c14873a0be5 => generated 0 bytes in 643 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:39.967816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c38b6ea-2bb2-4d82-ab40-10d7cec368b8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:39.968303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c38b6ea-2bb2-4d82-ab40-10d7cec368b8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:39.969696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c38b6ea-2bb2-4d82-ab40-10d7cec368b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:40.014914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0c38b6ea-2bb2-4d82-ab40-10d7cec368b8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:40.014914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0c38b6ea-2bb2-4d82-ab40-10d7cec368b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:40.044978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:40.177331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5263c42c-1e89-4b6b-988c-596349bde2cf None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:40.177853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5263c42c-1e89-4b6b-988c-596349bde2cf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:40.177939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5263c42c-1e89-4b6b-988c-596349bde2cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:40.217390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 858/1699] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:40.240207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c38b6ea-2bb2-4d82-ab40-10d7cec368b8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:40.247021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0c38b6ea-2bb2-4d82-ab40-10d7cec368b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:40.355113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5263c42c-1e89-4b6b-988c-596349bde2cf None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:40.356510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5263c42c-1e89-4b6b-988c-596349bde2cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:40.441057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c38b6ea-2bb2-4d82-ab40-10d7cec368b8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:40.453330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e4dab3-6b26-4610-bbab-b735f54d16f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:40.454244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e4dab3-6b26-4610-bbab-b735f54d16f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:40.454701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e4dab3-6b26-4610-bbab-b735f54d16f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:40.455125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e4dab3-6b26-4610-bbab-b735f54d16f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:40.477454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e4dab3-6b26-4610-bbab-b735f54d16f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:40.479246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e4dab3-6b26-4610-bbab-b735f54d16f6 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:40.495147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 859/1700] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:40.496265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e4dab3-6b26-4610-bbab-b735f54d16f6 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:40.507001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e4dab3-6b26-4610-bbab-b735f54d16f6 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:40.512677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248bc9fa-0c13-4990-832a-2ba2ee258a46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:40.513783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bc9fa-0c13-4990-832a-2ba2ee258a46 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:40.514287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bc9fa-0c13-4990-832a-2ba2ee258a46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:40.515247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bc9fa-0c13-4990-832a-2ba2ee258a46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:40.536797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248bc9fa-0c13-4990-832a-2ba2ee258a46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:40.538887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248bc9fa-0c13-4990-832a-2ba2ee258a46 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:40.569399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 842/1701] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:13:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 393 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:40.570005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 860/1702] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:40.716103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6fd27-7cac-4eff-b09d-6807f3f15afa None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:40.716292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6fd27-7cac-4eff-b09d-6807f3f15afa None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:40.716427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6fd27-7cac-4eff-b09d-6807f3f15afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:40.774963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-0ba6fd27-7cac-4eff-b09d-6807f3f15afa None None] MFA Rules not processed for user `b6ca9962541744bf9d67cc3e05721413`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:40.776204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0ba6fd27-7cac-4eff-b09d-6807f3f15afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:40.908251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 843/1703] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:40 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:40.919006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc6fb0d-5b5d-4563-9fe4-b2fd28ff964b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:40.919853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc6fb0d-5b5d-4563-9fe4-b2fd28ff964b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:40.920327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc6fb0d-5b5d-4563-9fe4-b2fd28ff964b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:40.943312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bbc6fb0d-5b5d-4563-9fe4-b2fd28ff964b None None] MFA Rules not processed for user `b6ca9962541744bf9d67cc3e05721413`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:40.945540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bbc6fb0d-5b5d-4563-9fe4-b2fd28ff964b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:40.954516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 861/1704] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:40 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:40.966548 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6fd27-7cac-4eff-b09d-6807f3f15afa None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:40.972312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0ba6fd27-7cac-4eff-b09d-6807f3f15afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:41.046636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc6fb0d-5b5d-4563-9fe4-b2fd28ff964b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:41.048874 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bbc6fb0d-5b5d-4563-9fe4-b2fd28ff964b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:41.146534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6fd27-7cac-4eff-b09d-6807f3f15afa None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:41.162291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:41.163214 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:41.163343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:41.165867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:41.185328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:41.188741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:41.213717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 844/1705] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:40 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:41.264409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc6fb0d-5b5d-4563-9fe4-b2fd28ff964b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:41.276965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de87f80-3d50-4667-a41b-976bb98b7bfc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:41.277776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de87f80-3d50-4667-a41b-976bb98b7bfc service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:41.278248 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de87f80-3d50-4667-a41b-976bb98b7bfc service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:41.278690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de87f80-3d50-4667-a41b-976bb98b7bfc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:41.293094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de87f80-3d50-4667-a41b-976bb98b7bfc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:41.295978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de87f80-3d50-4667-a41b-976bb98b7bfc service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:41.320130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 862/1706] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:13:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:13:41.429306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:41.433757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:41.467839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeee623-289f-4f7e-97a7-773e535daaf6 service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:41.506000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:41.507163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:41.507775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:41.508315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] PATH_INFO: `/v3/projects/ac2508877957498687af6f2e55b7b585` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:41.511572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac2508877957498687af6f2e55b7b585)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:41.514746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:41.700381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac2508877957498687af6f2e55b7b585', 'request_id': 'req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:41.804894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac2508877957498687af6f2e55b7b585', 'request_id': 'req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:41.814840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac2508877957498687af6f2e55b7b585', 'request_id': 'req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:41.855138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] The token cache is being invalidate because project ac2508877957498687af6f2e55b7b585 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:41.868187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:41.869483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1dbaf3c6-ae27-4e12-b09f-7dc9c51bdc06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:41.878383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 845/1707] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:41 2022] DELETE /identity/v3/projects/ac2508877957498687af6f2e55b7b585 => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:41.882281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de87f80-3d50-4667-a41b-976bb98b7bfc service swift] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:41.899508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de87f80-3d50-4667-a41b-976bb98b7bfc service swift] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:42.162496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:42.164589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:42.166726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:42.167288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] PATH_INFO: `/v3/users/a5bf61fe6763450890369a56e9425e39` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:42.168925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5bf61fe6763450890369a56e9425e39)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:42.171498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:42.456319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:42.544654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5bf61fe6763450890369a56e9425e39'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:42.546198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5bf61fe6763450890369a56e9425e39'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:42.600254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5bf61fe6763450890369a56e9425e39'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:42.730598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5bf61fe6763450890369a56e9425e39'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:42.742409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4a883e2f-85c6-41e2-9fd1-5f57c5bc5f64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5bf61fe6763450890369a56e9425e39'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:42.772241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 863/1708] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:41 2022] DELETE /identity/v3/users/a5bf61fe6763450890369a56e9425e39 => generated 0 bytes in 888 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:42.914854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3193b-588b-4b5c-a6db-b0d64647b961 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:42.915687 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3193b-588b-4b5c-a6db-b0d64647b961 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:42.916396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3193b-588b-4b5c-a6db-b0d64647b961 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:42.923788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 846/1709] 10.222.0.16 () {54 vars in 912 bytes} [Thu Aug 25 05:13:42 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:13:42.930796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bc3dae-c949-4414-a171-06030da2b341 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:42.931277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bc3dae-c949-4414-a171-06030da2b341 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:42.931769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bc3dae-c949-4414-a171-06030da2b341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:42.975499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f7bc3dae-c949-4414-a171-06030da2b341 None None] MFA Rules not processed for user `9220ba3637a542a382ad1d42ff1ff459`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:42.976772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f7bc3dae-c949-4414-a171-06030da2b341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:43.121710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 864/1710] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:13:42 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:43.149536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3193b-588b-4b5c-a6db-b0d64647b961 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:43.151688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ecb3193b-588b-4b5c-a6db-b0d64647b961 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:43.181451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3193b-588b-4b5c-a6db-b0d64647b961 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:43.204281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c527f0a7-d684-4a61-ba12-86b2a82da6f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:43.204281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c527f0a7-d684-4a61-ba12-86b2a82da6f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:43.204281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c527f0a7-d684-4a61-ba12-86b2a82da6f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:43.204281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c527f0a7-d684-4a61-ba12-86b2a82da6f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:43.251180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c527f0a7-d684-4a61-ba12-86b2a82da6f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:43.256710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c527f0a7-d684-4a61-ba12-86b2a82da6f0 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:43.276127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 847/1711] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:43 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:43.439647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:43.548027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bc3dae-c949-4414-a171-06030da2b341 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:43.549629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f7bc3dae-c949-4414-a171-06030da2b341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:43.626265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14e19f-2f0a-4c63-b38c-dbed5fce2ecc None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:43.626911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14e19f-2f0a-4c63-b38c-dbed5fce2ecc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:43.626911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14e19f-2f0a-4c63-b38c-dbed5fce2ecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:43.680448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bc3dae-c949-4414-a171-06030da2b341 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:43.703043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:43.703808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:43.704164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:43.704522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] PATH_INFO: `/v3/projects/99e2bf92350a413a8f593770a3abe5d2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:43.705895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=99e2bf92350a413a8f593770a3abe5d2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:43.707916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:43.724110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-0f14e19f-2f0a-4c63-b38c-dbed5fce2ecc None None] MFA Rules not processed for user `3326fcf3ccbc4a978e9e52e25456ad98`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:43.733267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0f14e19f-2f0a-4c63-b38c-dbed5fce2ecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:43.945669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '99e2bf92350a413a8f593770a3abe5d2', 'request_id': 'req-28fad8cb-1b92-45ae-809f-644d5dae8fe1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:43.975301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '99e2bf92350a413a8f593770a3abe5d2', 'request_id': 'req-28fad8cb-1b92-45ae-809f-644d5dae8fe1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:43.979287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 848/1712] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:43 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 361 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:43.984939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f14e19f-2f0a-4c63-b38c-dbed5fce2ecc None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:43.994532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0f14e19f-2f0a-4c63-b38c-dbed5fce2ecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:44.061987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '99e2bf92350a413a8f593770a3abe5d2', 'request_id': 'req-28fad8cb-1b92-45ae-809f-644d5dae8fe1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:44.087299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] The token cache is being invalidate because project 99e2bf92350a413a8f593770a3abe5d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:44.087795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:44.093911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-28fad8cb-1b92-45ae-809f-644d5dae8fe1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:44.100122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 865/1713] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:43 2022] DELETE /identity/v3/projects/99e2bf92350a413a8f593770a3abe5d2 => generated 0 bytes in 549 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:44.100122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e0b865-9cae-4dc8-b8c4-2e2ca609c05a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:44.101616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e0b865-9cae-4dc8-b8c4-2e2ca609c05a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:44.102309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e0b865-9cae-4dc8-b8c4-2e2ca609c05a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:44.193852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-91e0b865-9cae-4dc8-b8c4-2e2ca609c05a None None] MFA Rules not processed for user `d61cf1670a7a43519c7258cd467227b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:44.203994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-91e0b865-9cae-4dc8-b8c4-2e2ca609c05a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:44.247853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f14e19f-2f0a-4c63-b38c-dbed5fce2ecc None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:44.362901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e30d700-8b6c-490e-b6f0-92c0083296e2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:44.363541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e30d700-8b6c-490e-b6f0-92c0083296e2 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:44.363827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e30d700-8b6c-490e-b6f0-92c0083296e2 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:44.364421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e30d700-8b6c-490e-b6f0-92c0083296e2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:44.367330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6e30d700-8b6c-490e-b6f0-92c0083296e2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:44.403421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:44.471947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 866/1714] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:44 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 374 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:44.479480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e0b865-9cae-4dc8-b8c4-2e2ca609c05a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:44.479480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-91e0b865-9cae-4dc8-b8c4-2e2ca609c05a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:44.487594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e30d700-8b6c-490e-b6f0-92c0083296e2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:44.490668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e30d700-8b6c-490e-b6f0-92c0083296e2 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:44.506652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 849/1715] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:43 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:13:44.510052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af07b48-10eb-4cb9-ae93-f3fb61d6ce2d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:44.510052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af07b48-10eb-4cb9-ae93-f3fb61d6ce2d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:44.510052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af07b48-10eb-4cb9-ae93-f3fb61d6ce2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:44.614024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e0b865-9cae-4dc8-b8c4-2e2ca609c05a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:44.628069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:44.629154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:44.629966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:44.630762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] PATH_INFO: `/v3/projects/c212520a3f1145f8a89af8f197d83b76` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:44.632205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c212520a3f1145f8a89af8f197d83b76)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:44.634801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:44.643758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5af07b48-10eb-4cb9-ae93-f3fb61d6ce2d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:44.645160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5af07b48-10eb-4cb9-ae93-f3fb61d6ce2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:44.681284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 850/1716] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:44.689667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16599dd-246f-4dd3-9217-82e821331a88 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:44.690058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16599dd-246f-4dd3-9217-82e821331a88 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:44.691240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16599dd-246f-4dd3-9217-82e821331a88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:44.729063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e16599dd-246f-4dd3-9217-82e821331a88 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:44.730553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e16599dd-246f-4dd3-9217-82e821331a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:44.746609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 851/1717] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:44.747937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793b8a15-eec2-47ac-a5e3-4f8640c05547 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:44.748204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793b8a15-eec2-47ac-a5e3-4f8640c05547 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:44.748571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793b8a15-eec2-47ac-a5e3-4f8640c05547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:44.785096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-793b8a15-eec2-47ac-a5e3-4f8640c05547 None None] MFA Rules not processed for user `9220ba3637a542a382ad1d42ff1ff459`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:44.795562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-793b8a15-eec2-47ac-a5e3-4f8640c05547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:44.864311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c212520a3f1145f8a89af8f197d83b76', 'request_id': 'req-042cec3a-9353-4338-8c4c-d6b4cc372417'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:44.895567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c212520a3f1145f8a89af8f197d83b76', 'request_id': 'req-042cec3a-9353-4338-8c4c-d6b4cc372417'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:44.951712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 852/1718] 10.222.0.47 () {58 vars in 1015 bytes} [Thu Aug 25 05:13:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:44.953478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793b8a15-eec2-47ac-a5e3-4f8640c05547 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:44.993492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793b8a15-eec2-47ac-a5e3-4f8640c05547 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:45.011235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c212520a3f1145f8a89af8f197d83b76', 'request_id': 'req-042cec3a-9353-4338-8c4c-d6b4cc372417'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:45.026430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] The token cache is being invalidate because project c212520a3f1145f8a89af8f197d83b76 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:45.026733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:45.027415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:45.029118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 867/1719] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:44 2022] DELETE /identity/v3/projects/c212520a3f1145f8a89af8f197d83b76 => generated 0 bytes in 557 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:45.030974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:45.033719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:45.130059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:45.169036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6353dbe-e81e-4b86-98e9-cfaac58db8fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:45.172003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6353dbe-e81e-4b86-98e9-cfaac58db8fd admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:45.172421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6353dbe-e81e-4b86-98e9-cfaac58db8fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:45.172566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6353dbe-e81e-4b86-98e9-cfaac58db8fd admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:45.173239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6353dbe-e81e-4b86-98e9-cfaac58db8fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:45.173544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6353dbe-e81e-4b86-98e9-cfaac58db8fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:45.181411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6353dbe-e81e-4b86-98e9-cfaac58db8fd admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:45.218247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 853/1720] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:45.220230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835c789-0cea-4317-b42f-785208ef06c8 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:45.220526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835c789-0cea-4317-b42f-785208ef06c8 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:45.220856 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835c789-0cea-4317-b42f-785208ef06c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:45.240031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042cec3a-9353-4338-8c4c-d6b4cc372417 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:45.272297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:45.275795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:45.275795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:45.275795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:45.276113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1835c789-0cea-4317-b42f-785208ef06c8 None None] MFA Rules not processed for user `b6ca9962541744bf9d67cc3e05721413`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:45.277494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1835c789-0cea-4317-b42f-785208ef06c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:45.283276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:45.484028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:45.485689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:45.508939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 854/1721] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:45 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:45.510192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835c789-0cea-4317-b42f-785208ef06c8 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:45.512301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1835c789-0cea-4317-b42f-785208ef06c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:45.536126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 868/1722] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:45.537084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:45.545711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:45.594729 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb8413-61cd-4961-bfb6-260b1f68e18c service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:45.609020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:45.611979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:45.614358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:45.614358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] PATH_INFO: `/v3/users/df764242ae374ef1b119974d4dcbe0b1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:45.616686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df764242ae374ef1b119974d4dcbe0b1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:45.625365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:45.663793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835c789-0cea-4317-b42f-785208ef06c8 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:45.676267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:45.676267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:45.676564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:45.676778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:45.679400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:45.697664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:45.699153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:45.732973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 855/1723] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:45.734459 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:45.737994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:45.798187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a53c32d-2947-4e65-aa5a-d074fec94047 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:45.808315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:45.809438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:45.809801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:45.810145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] PATH_INFO: `/v3/projects/2683783126e649808c64d06244f531ac` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:45.812269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2683783126e649808c64d06244f531ac)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:45.815165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:45.971721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df764242ae374ef1b119974d4dcbe0b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:45.980567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df764242ae374ef1b119974d4dcbe0b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.026296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2683783126e649808c64d06244f531ac', 'request_id': 'req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.029881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df764242ae374ef1b119974d4dcbe0b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.119172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:46.147092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2683783126e649808c64d06244f531ac', 'request_id': 'req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.154174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2683783126e649808c64d06244f531ac', 'request_id': 'req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.198091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] The token cache is being invalidate because project 2683783126e649808c64d06244f531ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:46.202023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.207221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.210234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 856/1724] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:45 2022] DELETE /identity/v3/projects/2683783126e649808c64d06244f531ac => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:46.211337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:46.227731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:46.278343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df764242ae374ef1b119974d4dcbe0b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.290645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-602b09f0-083d-4f34-ad2e-d1314ce737c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df764242ae374ef1b119974d4dcbe0b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:46.309104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 869/1725] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:45 2022] DELETE /identity/v3/users/df764242ae374ef1b119974d4dcbe0b1 => generated 0 bytes in 772 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:46.312668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ab5b5f-0fc7-4c40-a3c0-6d419ab8f846 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:46.315130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ab5b5f-0fc7-4c40-a3c0-6d419ab8f846 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:46.315734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ab5b5f-0fc7-4c40-a3c0-6d419ab8f846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:46.341217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:46.371978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-52ab5b5f-0fc7-4c40-a3c0-6d419ab8f846 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:46.393414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-52ab5b5f-0fc7-4c40-a3c0-6d419ab8f846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:46.475725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef6b39-02de-4da4-88b0-1a320ac7ce1e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:46.681098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 870/1726] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:46.682667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a515de2-ad33-4896-aff6-76f30a4eaaba None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:46.682964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a515de2-ad33-4896-aff6-76f30a4eaaba None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:46.683245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a515de2-ad33-4896-aff6-76f30a4eaaba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:46.757773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3a515de2-ad33-4896-aff6-76f30a4eaaba None None] MFA Rules not processed for user `ec39d995899c4d43bb4f8ffd9e6d2ef3`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:46.760317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3a515de2-ad33-4896-aff6-76f30a4eaaba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:46.811940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a4d2c9-7ff1-408f-b1d0-80780c8ee325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:46.812591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4d2c9-7ff1-408f-b1d0-80780c8ee325 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:46.812885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4d2c9-7ff1-408f-b1d0-80780c8ee325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:46.813229 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a4d2c9-7ff1-408f-b1d0-80780c8ee325 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:46.813974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a4d2c9-7ff1-408f-b1d0-80780c8ee325 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:46.835620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a4d2c9-7ff1-408f-b1d0-80780c8ee325 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:47.063305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 857/1727] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:46 2022] POST /identity/v3/projects => generated 381 bytes in 853 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:47.069547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb61767-c92b-4273-b56a-51b50a233b23 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:47.077487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb61767-c92b-4273-b56a-51b50a233b23 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:47.078595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb61767-c92b-4273-b56a-51b50a233b23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:47.095287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 871/1728] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 414 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:47.100166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a515de2-ad33-4896-aff6-76f30a4eaaba None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:47.153327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a515de2-ad33-4896-aff6-76f30a4eaaba None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:47.156359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2eb61767-c92b-4273-b56a-51b50a233b23 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:47.157723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2eb61767-c92b-4273-b56a-51b50a233b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:47.369886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 858/1729] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:47.374541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb61767-c92b-4273-b56a-51b50a233b23 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:47.410753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb61767-c92b-4273-b56a-51b50a233b23 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:47.424045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212a1e88-b4c6-4ed6-8783-f3bde6c81a7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:47.426303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212a1e88-b4c6-4ed6-8783-f3bde6c81a7d admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:47.428164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212a1e88-b4c6-4ed6-8783-f3bde6c81a7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:47.429128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212a1e88-b4c6-4ed6-8783-f3bde6c81a7d admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:47.430451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212a1e88-b4c6-4ed6-8783-f3bde6c81a7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:47.435485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212a1e88-b4c6-4ed6-8783-f3bde6c81a7d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:47.436964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:47.438270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:47.438573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:47.438854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:47.443013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:47.443350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:47.448503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:47.511016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 859/1730] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:13:47.512179 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:47.578025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527ffc3a-ea3d-4aef-afdb-c9ce1c82a223 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:47.603761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:47.604400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:47.604733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:47.611806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:47.614524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:47.631209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 872/1731] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:47 2022] POST /identity/v3/users => generated 291 bytes in 531 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:47.632488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212a1e88-b4c6-4ed6-8783-f3bde6c81a7d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:47.632604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:47.652951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212a1e88-b4c6-4ed6-8783-f3bde6c81a7d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:47.663788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0bee9c-bf02-4ec2-9b94-de79582ecf73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:47.665674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0bee9c-bf02-4ec2-9b94-de79582ecf73 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:47.665674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0bee9c-bf02-4ec2-9b94-de79582ecf73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:47.665674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0bee9c-bf02-4ec2-9b94-de79582ecf73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:47.670885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0bee9c-bf02-4ec2-9b94-de79582ecf73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:47.670885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0bee9c-bf02-4ec2-9b94-de79582ecf73 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:47.724378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 873/1732] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:47 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:47.726463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0bee9c-bf02-4ec2-9b94-de79582ecf73 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:47.760582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0bee9c-bf02-4ec2-9b94-de79582ecf73 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:47.786414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbd91d9-9c15-46e9-bafc-52c9f4acd023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:47.787844 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbd91d9-9c15-46e9-bafc-52c9f4acd023 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:47.796848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbd91d9-9c15-46e9-bafc-52c9f4acd023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:47.796848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbd91d9-9c15-46e9-bafc-52c9f4acd023 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:47.796848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbd91d9-9c15-46e9-bafc-52c9f4acd023 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:47.804820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbd91d9-9c15-46e9-bafc-52c9f4acd023 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:47.824652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 860/1733] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:47 2022] POST /identity/v3/projects => generated 379 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:47.825654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:47.829972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:47.856024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 874/1734] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:47 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:47.881877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbd91d9-9c15-46e9-bafc-52c9f4acd023 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:47.943256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbd91d9-9c15-46e9-bafc-52c9f4acd023 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:48.124623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c612bf-a0dd-4323-9e59-3d5c40523979 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:48.154290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:48.155468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:48.155753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:48.166858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] PATH_INFO: `/v3/projects/32057f9a02074fdabcf858de7e733bc0` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:48.170296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=32057f9a02074fdabcf858de7e733bc0)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:48.179745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3258687c-435e-4c04-99fb-249e893b44aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:48.180255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:48.182213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258687c-435e-4c04-99fb-249e893b44aa admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:48.182503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258687c-435e-4c04-99fb-249e893b44aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:48.182779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3258687c-435e-4c04-99fb-249e893b44aa admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:48.183424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3258687c-435e-4c04-99fb-249e893b44aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:48.198767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3258687c-435e-4c04-99fb-249e893b44aa admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:48.303201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 875/1735] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:47 2022] POST /identity/v3/users => generated 290 bytes in 447 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:48.304125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3258687c-435e-4c04-99fb-249e893b44aa admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:48.363984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3258687c-435e-4c04-99fb-249e893b44aa admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:48.396521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32057f9a02074fdabcf858de7e733bc0', 'request_id': 'req-15371dcd-d347-4356-941f-b9ff37cf13ee'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:48.601539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228f94ab-cbc4-49e7-ab3b-2dc88e5d93ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:48.602535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f94ab-cbc4-49e7-ab3b-2dc88e5d93ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:48.603560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f94ab-cbc4-49e7-ab3b-2dc88e5d93ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:48.604008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f94ab-cbc4-49e7-ab3b-2dc88e5d93ac admin admin] PATH_INFO: `/v3/projects/eed65c626fa94427924203ac6546c3be/users/17e369e4e2e3475099521852c727fe59/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:48.621520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32057f9a02074fdabcf858de7e733bc0', 'request_id': 'req-15371dcd-d347-4356-941f-b9ff37cf13ee'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:48.630590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32057f9a02074fdabcf858de7e733bc0', 'request_id': 'req-15371dcd-d347-4356-941f-b9ff37cf13ee'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:48.639639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228f94ab-cbc4-49e7-ab3b-2dc88e5d93ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=eed65c626fa94427924203ac6546c3be, user_id=17e369e4e2e3475099521852c727fe59, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:48.647919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228f94ab-cbc4-49e7-ab3b-2dc88e5d93ac admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:48.667766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] The token cache is being invalidate because project 32057f9a02074fdabcf858de7e733bc0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:48.669690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:48.671358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:48.681940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 861/1736] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:47 2022] DELETE /identity/v3/projects/32057f9a02074fdabcf858de7e733bc0 => generated 0 bytes in 857 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:48.682678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:48.688818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:48.690099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:48.734636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15371dcd-d347-4356-941f-b9ff37cf13ee admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:48.742444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 876/1737] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:48 2022] PUT /identity/v3/projects/eed65c626fa94427924203ac6546c3be/users/17e369e4e2e3475099521852c727fe59/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:48.744334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956962cc-a43b-4018-83aa-8d3b2a0d3afa None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:48.745634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956962cc-a43b-4018-83aa-8d3b2a0d3afa None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:48.745634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956962cc-a43b-4018-83aa-8d3b2a0d3afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:48.802295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-956962cc-a43b-4018-83aa-8d3b2a0d3afa None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:48.811566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-956962cc-a43b-4018-83aa-8d3b2a0d3afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:48.981473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cc6f58-c5e1-4703-89a6-e7f9bcebe808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:48.983309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cc6f58-c5e1-4703-89a6-e7f9bcebe808 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:48.983592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cc6f58-c5e1-4703-89a6-e7f9bcebe808 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:48.983871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cc6f58-c5e1-4703-89a6-e7f9bcebe808 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:48.988233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cc6f58-c5e1-4703-89a6-e7f9bcebe808 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:48.990016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cc6f58-c5e1-4703-89a6-e7f9bcebe808 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:49.000367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 877/1738] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:49.023731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeaff37-03ea-461b-914c-98adaeb4e003 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.023731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeaff37-03ea-461b-914c-98adaeb4e003 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.023731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeaff37-03ea-461b-914c-98adaeb4e003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.040741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 862/1739] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:48 2022] GET /identity/v3/roles => generated 1420 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:49.054020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f76148-3b2f-402e-bc6a-31b739605a50 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.054406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f76148-3b2f-402e-bc6a-31b739605a50 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.054851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f76148-3b2f-402e-bc6a-31b739605a50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.153107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-08f76148-3b2f-402e-bc6a-31b739605a50 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:49.159460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-08f76148-3b2f-402e-bc6a-31b739605a50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:49.195058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-caeaff37-03ea-461b-914c-98adaeb4e003 None None] MFA Rules not processed for user `17e369e4e2e3475099521852c727fe59`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:49.195475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 863/1740] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:49.196496 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f76148-3b2f-402e-bc6a-31b739605a50 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:49.207813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:49.208661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-caeaff37-03ea-461b-914c-98adaeb4e003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:49.233453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f76148-3b2f-402e-bc6a-31b739605a50 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:49.252470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2871a184-081a-46b2-8235-39243798312e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:49.253099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2871a184-081a-46b2-8235-39243798312e admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.253385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2871a184-081a-46b2-8235-39243798312e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.257286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2871a184-081a-46b2-8235-39243798312e admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.258111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2871a184-081a-46b2-8235-39243798312e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:49.263616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2871a184-081a-46b2-8235-39243798312e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:49.306858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 864/1741] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:49 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:49.308634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2871a184-081a-46b2-8235-39243798312e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:49.341394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2871a184-081a-46b2-8235-39243798312e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:49.362863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:49.363498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.363932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.364218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.364880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:13:49.365187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:49.366836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:49.416052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 865/1742] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:13:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:13:49.418531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:49.452818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6876895d-eca1-40a3-8f40-6da82af0227e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:49.475708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:49.476544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.476833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.477110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] PATH_INFO: `/v3/projects/dd6717beff0f4843bc9200f907e46dbc/users/525f10462381466397878761517ffcfc/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.477929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 878/1743] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:49 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 477 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:49.479156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeaff37-03ea-461b-914c-98adaeb4e003 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:49.526447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd6717beff0f4843bc9200f907e46dbc, user_id=525f10462381466397878761517ffcfc, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:49.529151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:49.541673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeaff37-03ea-461b-914c-98adaeb4e003 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:49.557533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541dd16a-4d5d-49f7-90c1-3d0e4181ec82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:49.558369 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541dd16a-4d5d-49f7-90c1-3d0e4181ec82 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.558685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541dd16a-4d5d-49f7-90c1-3d0e4181ec82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.558967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541dd16a-4d5d-49f7-90c1-3d0e4181ec82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.561132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541dd16a-4d5d-49f7-90c1-3d0e4181ec82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:49.564821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541dd16a-4d5d-49f7-90c1-3d0e4181ec82 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:49.656475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 866/1744] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:49 2022] PUT /identity/v3/projects/dd6717beff0f4843bc9200f907e46dbc/users/525f10462381466397878761517ffcfc/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:49.657615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:49.663960 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:49.768976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 879/1745] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:49 2022] POST /identity/v3/projects => generated 409 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:49.770585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367e4ec7-998c-4d72-91a7-3a200bacb5bb None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.770877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367e4ec7-998c-4d72-91a7-3a200bacb5bb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.771151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367e4ec7-998c-4d72-91a7-3a200bacb5bb None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.779592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 880/1746] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:13:49 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:13:49.780614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd8711d-731d-4df1-91e9-1c4de215ab6c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.780858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd8711d-731d-4df1-91e9-1c4de215ab6c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.781251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd8711d-731d-4df1-91e9-1c4de215ab6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.843814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a2d213-7558-4a3a-9d36-01fdd3e6519e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:49.889566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ddd8711d-731d-4df1-91e9-1c4de215ab6c None None] MFA Rules not processed for user `525f10462381466397878761517ffcfc`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:49.892255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ddd8711d-731d-4df1-91e9-1c4de215ab6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:49.984483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453ab53-ddd8-4960-aee5-e252f29e388f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:49.992034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453ab53-ddd8-4960-aee5-e252f29e388f admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:49.992034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453ab53-ddd8-4960-aee5-e252f29e388f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:49.992034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453ab53-ddd8-4960-aee5-e252f29e388f admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:49.992034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453ab53-ddd8-4960-aee5-e252f29e388f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:49.992034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453ab53-ddd8-4960-aee5-e252f29e388f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:50.097844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 867/1747] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:49 2022] POST /identity/v3/projects => generated 381 bytes in 441 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:50.100698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453ab53-ddd8-4960-aee5-e252f29e388f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:50.118762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 881/1748] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:49 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:50.120167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4365d2ac-2f5b-4e57-9902-6365239a1211 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:50.120456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4365d2ac-2f5b-4e57-9902-6365239a1211 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:50.120727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4365d2ac-2f5b-4e57-9902-6365239a1211 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:50.217491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4365d2ac-2f5b-4e57-9902-6365239a1211 None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:50.226800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4365d2ac-2f5b-4e57-9902-6365239a1211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:50.339926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453ab53-ddd8-4960-aee5-e252f29e388f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:50.353401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d87628-410c-4af3-b4de-7cf89cfe653a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:50.355620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d87628-410c-4af3-b4de-7cf89cfe653a admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:50.372759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d87628-410c-4af3-b4de-7cf89cfe653a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:50.373859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d87628-410c-4af3-b4de-7cf89cfe653a admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:50.376944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d87628-410c-4af3-b4de-7cf89cfe653a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:50.409704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 882/1749] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:13:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:50.410518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4365d2ac-2f5b-4e57-9902-6365239a1211 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:50.413253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d87628-410c-4af3-b4de-7cf89cfe653a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:50.434111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4365d2ac-2f5b-4e57-9902-6365239a1211 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:50.449197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80125e51-0ace-4418-9663-a20f6e670672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:50.453481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80125e51-0ace-4418-9663-a20f6e670672 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:50.453481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80125e51-0ace-4418-9663-a20f6e670672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:50.453481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80125e51-0ace-4418-9663-a20f6e670672 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:50.453481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80125e51-0ace-4418-9663-a20f6e670672 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:50.466213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80125e51-0ace-4418-9663-a20f6e670672 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:50.594919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:50.651404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 868/1750] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:50 2022] POST /identity/v3/users => generated 305 bytes in 553 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:50.652435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d87628-410c-4af3-b4de-7cf89cfe653a admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:50.676740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d87628-410c-4af3-b4de-7cf89cfe653a admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:50.687090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:50.688704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:50.689013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:50.689295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:50.690217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:50.710392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 883/1751] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:50 2022] POST /identity/v3/users => generated 297 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:50.711276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80125e51-0ace-4418-9663-a20f6e670672 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:50.721534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:50.779566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80125e51-0ace-4418-9663-a20f6e670672 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:50.797455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d35187-56ab-442c-acd2-17011ee61aaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:50.798651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d35187-56ab-442c-acd2-17011ee61aaf admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:50.799085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d35187-56ab-442c-acd2-17011ee61aaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:50.799368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d35187-56ab-442c-acd2-17011ee61aaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:50.800415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d35187-56ab-442c-acd2-17011ee61aaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:50.819505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d35187-56ab-442c-acd2-17011ee61aaf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:50.857535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 884/1752] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:50 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:50.858490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d35187-56ab-442c-acd2-17011ee61aaf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:50.882553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d35187-56ab-442c-acd2-17011ee61aaf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:50.891281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ae0c1-897b-4166-83c7-c5f4bc41caf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:50.891971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ae0c1-897b-4166-83c7-c5f4bc41caf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:50.892311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ae0c1-897b-4166-83c7-c5f4bc41caf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:50.892835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680ae0c1-897b-4166-83c7-c5f4bc41caf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:50.893925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680ae0c1-897b-4166-83c7-c5f4bc41caf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:50.895938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680ae0c1-897b-4166-83c7-c5f4bc41caf1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:50.941838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 869/1753] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:50 2022] POST /identity/v3/projects => generated 379 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:50.946213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:50.952431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:50.959085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 885/1754] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:50 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:50.963020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ae0c1-897b-4166-83c7-c5f4bc41caf1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:51.011348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680ae0c1-897b-4166-83c7-c5f4bc41caf1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:51.231650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b26dfe5-1ded-4dbc-8bb2-9cd99275341b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:51.232276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b26dfe5-1ded-4dbc-8bb2-9cd99275341b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:51.233127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b26dfe5-1ded-4dbc-8bb2-9cd99275341b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:51.233438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b26dfe5-1ded-4dbc-8bb2-9cd99275341b admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:51.234756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b26dfe5-1ded-4dbc-8bb2-9cd99275341b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:51.236435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b26dfe5-1ded-4dbc-8bb2-9cd99275341b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:51.264738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 886/1755] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:50 2022] GET /identity/v3/roles => generated 1420 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:51.264945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b26dfe5-1ded-4dbc-8bb2-9cd99275341b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:51.285762 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4bc8b2-2b67-4f53-b193-ec15d835ebfe admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:51.290940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b26dfe5-1ded-4dbc-8bb2-9cd99275341b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:51.300052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397b7991-4f9c-4ab1-b6c7-443d145e9536 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:51.302860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397b7991-4f9c-4ab1-b6c7-443d145e9536 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:51.303291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397b7991-4f9c-4ab1-b6c7-443d145e9536 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:51.303777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397b7991-4f9c-4ab1-b6c7-443d145e9536 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:51.308308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00123bb-54fb-4d18-a37a-1b8b4225e10e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:51.309151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00123bb-54fb-4d18-a37a-1b8b4225e10e admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:51.309485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00123bb-54fb-4d18-a37a-1b8b4225e10e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:51.309784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00123bb-54fb-4d18-a37a-1b8b4225e10e admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:51.310669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00123bb-54fb-4d18-a37a-1b8b4225e10e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:51.315183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00123bb-54fb-4d18-a37a-1b8b4225e10e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:51.330777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397b7991-4f9c-4ab1-b6c7-443d145e9536 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:51.334479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397b7991-4f9c-4ab1-b6c7-443d145e9536 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:51.370855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 870/1756] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:51.375137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397b7991-4f9c-4ab1-b6c7-443d145e9536 service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:51.409515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397b7991-4f9c-4ab1-b6c7-443d145e9536 service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:51.439801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc00ad7-d016-4b01-91d3-fdd0bc441432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:51.441039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc00ad7-d016-4b01-91d3-fdd0bc441432 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:51.441514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc00ad7-d016-4b01-91d3-fdd0bc441432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:51.441954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc00ad7-d016-4b01-91d3-fdd0bc441432 admin admin] PATH_INFO: `/v3/projects/1d1d7fde888a49b0be635f9dde6002dd/users/c9c0d2ce838b4c939bdf1bf15aa39a11/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:51.494609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc00ad7-d016-4b01-91d3-fdd0bc441432 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d1d7fde888a49b0be635f9dde6002dd, user_id=c9c0d2ce838b4c939bdf1bf15aa39a11, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:51.497333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc00ad7-d016-4b01-91d3-fdd0bc441432 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:51.560139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 887/1757] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:51 2022] POST /identity/v3/users => generated 296 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:51.561078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00123bb-54fb-4d18-a37a-1b8b4225e10e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:51.601670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00123bb-54fb-4d18-a37a-1b8b4225e10e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:51.608308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f33733-d62e-49bf-b4a2-73dd9f75b796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:51.615289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f33733-d62e-49bf-b4a2-73dd9f75b796 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:51.619255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f33733-d62e-49bf-b4a2-73dd9f75b796 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:51.619684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f33733-d62e-49bf-b4a2-73dd9f75b796 admin admin] PATH_INFO: `/v3/projects/3d75667a7ba740daa9512c0d60782883/users/08e3871b74574ad99ce2cf382aa7a69e/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:51.668647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f33733-d62e-49bf-b4a2-73dd9f75b796 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d75667a7ba740daa9512c0d60782883, user_id=08e3871b74574ad99ce2cf382aa7a69e, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:51.672099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f33733-d62e-49bf-b4a2-73dd9f75b796 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:51.687127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 871/1758] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:51 2022] PUT /identity/v3/projects/1d1d7fde888a49b0be635f9dde6002dd/users/c9c0d2ce838b4c939bdf1bf15aa39a11/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:51.691578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc00ad7-d016-4b01-91d3-fdd0bc441432 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:51.817318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 888/1759] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:51 2022] PUT /identity/v3/projects/3d75667a7ba740daa9512c0d60782883/users/08e3871b74574ad99ce2cf382aa7a69e/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:51.817318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f33733-d62e-49bf-b4a2-73dd9f75b796 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:51.892661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc00ad7-d016-4b01-91d3-fdd0bc441432 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:51.994164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f33733-d62e-49bf-b4a2-73dd9f75b796 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:52.012500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e49d89d-a7f9-42b3-9c46-3da3884ed2be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:52.013255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e49d89d-a7f9-42b3-9c46-3da3884ed2be admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:52.013460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e49d89d-a7f9-42b3-9c46-3da3884ed2be admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:52.019547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e49d89d-a7f9-42b3-9c46-3da3884ed2be admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:52.020508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e49d89d-a7f9-42b3-9c46-3da3884ed2be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:52.022357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e49d89d-a7f9-42b3-9c46-3da3884ed2be admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:52.062965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01be2bed-fe5f-4bef-9e20-09cef286bac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:52.063836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01be2bed-fe5f-4bef-9e20-09cef286bac0 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:52.064131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01be2bed-fe5f-4bef-9e20-09cef286bac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:52.064428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01be2bed-fe5f-4bef-9e20-09cef286bac0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:52.065430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01be2bed-fe5f-4bef-9e20-09cef286bac0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:52.066820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01be2bed-fe5f-4bef-9e20-09cef286bac0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:52.077275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 889/1760] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:51 2022] GET /identity/v3/roles => generated 1420 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:52.080222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6c0b16-75ad-4d78-a48f-f11eb45c1af0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:52.080442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6c0b16-75ad-4d78-a48f-f11eb45c1af0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:52.080725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6c0b16-75ad-4d78-a48f-f11eb45c1af0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:52.124854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 872/1761] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:51 2022] GET /identity/v3/roles => generated 1420 bytes in 437 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:52.128529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01be2bed-fe5f-4bef-9e20-09cef286bac0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:52.174829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01be2bed-fe5f-4bef-9e20-09cef286bac0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:52.209148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0ecfc4-0ba5-4558-8be9-985cc32c0ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:52.214435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1c6c0b16-75ad-4d78-a48f-f11eb45c1af0 None None] MFA Rules not processed for user `08e3871b74574ad99ce2cf382aa7a69e`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:52.216772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0ecfc4-0ba5-4558-8be9-985cc32c0ebe admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:52.218094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1c6c0b16-75ad-4d78-a48f-f11eb45c1af0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:52.220785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0ecfc4-0ba5-4558-8be9-985cc32c0ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:52.221956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0ecfc4-0ba5-4558-8be9-985cc32c0ebe admin admin] PATH_INFO: `/v3/projects/1d1d7fde888a49b0be635f9dde6002dd/users/c9c0d2ce838b4c939bdf1bf15aa39a11/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:52.231014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0ecfc4-0ba5-4558-8be9-985cc32c0ebe admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d1d7fde888a49b0be635f9dde6002dd, user_id=c9c0d2ce838b4c939bdf1bf15aa39a11, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:52.249971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0ecfc4-0ba5-4558-8be9-985cc32c0ebe admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:52.266264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:52.299684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 873/1762] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:52 2022] PUT /identity/v3/projects/1d1d7fde888a49b0be635f9dde6002dd/users/c9c0d2ce838b4c939bdf1bf15aa39a11/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:52.300696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0ecfc4-0ba5-4558-8be9-985cc32c0ebe admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:52.456393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 890/1763] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:52 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 378 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:52.457727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f78b863-cd26-42e6-b73b-514a2f8d0f17 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:52.458041 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f78b863-cd26-42e6-b73b-514a2f8d0f17 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:52.463232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f78b863-cd26-42e6-b73b-514a2f8d0f17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:52.537543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0ecfc4-0ba5-4558-8be9-985cc32c0ebe admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:52.575456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4061a-ea6f-431e-b8c3-b06ad252f4da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:52.579010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4061a-ea6f-431e-b8c3-b06ad252f4da admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:52.579010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4061a-ea6f-431e-b8c3-b06ad252f4da admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:52.579010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4061a-ea6f-431e-b8c3-b06ad252f4da admin admin] PATH_INFO: `/v3/projects/98e91b1afc0f4ec89bc81ca41f1ff785/users/e2fbfb2883ab404eb98a2263b9f018ea/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:52.581492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5f78b863-cd26-42e6-b73b-514a2f8d0f17 None None] MFA Rules not processed for user `c9c0d2ce838b4c939bdf1bf15aa39a11`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:52.583479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5f78b863-cd26-42e6-b73b-514a2f8d0f17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:52.635081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c4061a-ea6f-431e-b8c3-b06ad252f4da admin admin] RBAC: Authorizing `identity:create_grant(project_id=98e91b1afc0f4ec89bc81ca41f1ff785, user_id=e2fbfb2883ab404eb98a2263b9f018ea, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:52.652206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c4061a-ea6f-431e-b8c3-b06ad252f4da admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:52.740730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 874/1764] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:52 2022] PUT /identity/v3/projects/98e91b1afc0f4ec89bc81ca41f1ff785/users/e2fbfb2883ab404eb98a2263b9f018ea/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:52.743634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4061a-ea6f-431e-b8c3-b06ad252f4da admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:52.768238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 891/1765] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:52 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:52.769703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f78b863-cd26-42e6-b73b-514a2f8d0f17 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:52.823198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f78b863-cd26-42e6-b73b-514a2f8d0f17 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:52.966138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:52.966138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:52.966138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:52.966138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:52.966421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:52.966421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:53.005315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 892/1766] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:52 2022] GET /identity/v3/roles => generated 1420 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:53.007054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:53.012520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4061a-ea6f-431e-b8c3-b06ad252f4da admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:53.018791 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:53.021703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452e2a8a-a469-433c-8445-d10b3b62e3f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:53.022665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e2a8a-a469-433c-8445-d10b3b62e3f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:53.023092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e2a8a-a469-433c-8445-d10b3b62e3f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:53.024475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452e2a8a-a469-433c-8445-d10b3b62e3f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:53.024475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452e2a8a-a469-433c-8445-d10b3b62e3f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:53.031068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452e2a8a-a469-433c-8445-d10b3b62e3f9 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:53.157333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a441831-b02d-426e-8eeb-4c579708e413 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:53.165719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9102e9-64c6-4b5a-9b38-b5996cf900b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:53.168122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9102e9-64c6-4b5a-9b38-b5996cf900b1 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:53.168710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9102e9-64c6-4b5a-9b38-b5996cf900b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:53.169053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9102e9-64c6-4b5a-9b38-b5996cf900b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:53.174348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 875/1767] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:13:52 2022] POST /identity/v3/projects => generated 407 bytes in 430 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:53.174348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452e2a8a-a469-433c-8445-d10b3b62e3f9 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:53.374274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452e2a8a-a469-433c-8445-d10b3b62e3f9 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:53.393218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e439f0d6-e06d-4d19-88f2-9d86513fb447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:53.394056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e439f0d6-e06d-4d19-88f2-9d86513fb447 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:53.395038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e439f0d6-e06d-4d19-88f2-9d86513fb447 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:53.395038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e439f0d6-e06d-4d19-88f2-9d86513fb447 admin admin] PATH_INFO: `/v3/projects/98e91b1afc0f4ec89bc81ca41f1ff785/users/e2fbfb2883ab404eb98a2263b9f018ea/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:53.396839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e439f0d6-e06d-4d19-88f2-9d86513fb447 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98e91b1afc0f4ec89bc81ca41f1ff785, user_id=e2fbfb2883ab404eb98a2263b9f018ea, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:53.398858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9102e9-64c6-4b5a-9b38-b5996cf900b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:53.400356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9102e9-64c6-4b5a-9b38-b5996cf900b1 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:53.404936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e439f0d6-e06d-4d19-88f2-9d86513fb447 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:53.423267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:53.436052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 876/1768] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:53 2022] PUT /identity/v3/projects/98e91b1afc0f4ec89bc81ca41f1ff785/users/e2fbfb2883ab404eb98a2263b9f018ea/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:53.437564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e439f0d6-e06d-4d19-88f2-9d86513fb447 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:53.446876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 893/1769] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:53 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:53.454321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae1e2b0-9923-4ea2-8d06-65b6af7e4c2e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:53.454647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae1e2b0-9923-4ea2-8d06-65b6af7e4c2e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:53.458140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae1e2b0-9923-4ea2-8d06-65b6af7e4c2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:53.574420 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5ae1e2b0-9923-4ea2-8d06-65b6af7e4c2e None None] MFA Rules not processed for user `e2fbfb2883ab404eb98a2263b9f018ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:53.576730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5ae1e2b0-9923-4ea2-8d06-65b6af7e4c2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:53.608959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e439f0d6-e06d-4d19-88f2-9d86513fb447 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:53.622099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f6d1fb-1f4a-4194-9f1e-336de6ceb2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:53.622741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f6d1fb-1f4a-4194-9f1e-336de6ceb2ac admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:53.625733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f6d1fb-1f4a-4194-9f1e-336de6ceb2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:53.625733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f6d1fb-1f4a-4194-9f1e-336de6ceb2ac admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:53.625733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f6d1fb-1f4a-4194-9f1e-336de6ceb2ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:53.631114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f6d1fb-1f4a-4194-9f1e-336de6ceb2ac admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:53.759706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 894/1770] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:53 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:53.774883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae1e2b0-9923-4ea2-8d06-65b6af7e4c2e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:53.805670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 877/1771] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:13:53 2022] POST /identity/v3/users => generated 310 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:13:53.809907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f6d1fb-1f4a-4194-9f1e-336de6ceb2ac admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:53.845119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f6d1fb-1f4a-4194-9f1e-336de6ceb2ac admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:53.850816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:53.851512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:53.851800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:53.852127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:53.852917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:53.854844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:53.899796 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 878/1772] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:53 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:53.901062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:53.917725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae1e2b0-9923-4ea2-8d06-65b6af7e4c2e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:53.922750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:53.929895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3442fbc8-9973-49b6-a5a7-9530d56658d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:53.930803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442fbc8-9973-49b6-a5a7-9530d56658d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:53.931505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442fbc8-9973-49b6-a5a7-9530d56658d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:53.931884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3442fbc8-9973-49b6-a5a7-9530d56658d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:53.938786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1a5411-6f5f-4be9-8327-6190f69fdb7c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:53.953580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cd7780-4720-4c94-b36f-02177aa5a9f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:53.954517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cd7780-4720-4c94-b36f-02177aa5a9f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:53.954868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cd7780-4720-4c94-b36f-02177aa5a9f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:53.955199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cd7780-4720-4c94-b36f-02177aa5a9f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:53.976659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3442fbc8-9973-49b6-a5a7-9530d56658d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:53.978285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3442fbc8-9973-49b6-a5a7-9530d56658d5 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:54.027432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 895/1773] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:53 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:54.028431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3442fbc8-9973-49b6-a5a7-9530d56658d5 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:54.059027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3442fbc8-9973-49b6-a5a7-9530d56658d5 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:54.072699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56321d8-4049-485e-b4ac-e6b30301eac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:54.073876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56321d8-4049-485e-b4ac-e6b30301eac5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:54.074176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56321d8-4049-485e-b4ac-e6b30301eac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:54.074461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56321d8-4049-485e-b4ac-e6b30301eac5 admin admin] PATH_INFO: `/v3/projects/652bc65c53ff4d53b155f229bd0c25e9/users/95b2bf0bbc18466081776c9dda55ac30/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:54.149522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56321d8-4049-485e-b4ac-e6b30301eac5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=652bc65c53ff4d53b155f229bd0c25e9, user_id=95b2bf0bbc18466081776c9dda55ac30, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:54.152394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56321d8-4049-485e-b4ac-e6b30301eac5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:54.170204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cd7780-4720-4c94-b36f-02177aa5a9f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:54.171956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cd7780-4720-4c94-b36f-02177aa5a9f2 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:54.216428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 879/1774] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:53 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:54.259896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 896/1775] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:54 2022] PUT /identity/v3/projects/652bc65c53ff4d53b155f229bd0c25e9/users/95b2bf0bbc18466081776c9dda55ac30/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:54.260490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cd7780-4720-4c94-b36f-02177aa5a9f2 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:54.293384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cd7780-4720-4c94-b36f-02177aa5a9f2 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:54.570474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b1beb4-964f-44a9-aca0-97d86885b27e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:54.571986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b1beb4-964f-44a9-aca0-97d86885b27e admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:54.572520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b1beb4-964f-44a9-aca0-97d86885b27e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:54.572995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b1beb4-964f-44a9-aca0-97d86885b27e admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:54.573900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b1beb4-964f-44a9-aca0-97d86885b27e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:54.575797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b1beb4-964f-44a9-aca0-97d86885b27e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:54.606782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 880/1776] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:13:54 2022] GET /identity/v3/roles => generated 1420 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:13:54.607950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56321d8-4049-485e-b4ac-e6b30301eac5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:54.631221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56321d8-4049-485e-b4ac-e6b30301eac5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:54.649336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdf9739-b028-4a75-921d-a822251c8011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:54.651356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdf9739-b028-4a75-921d-a822251c8011 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:54.652682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdf9739-b028-4a75-921d-a822251c8011 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:54.652945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdf9739-b028-4a75-921d-a822251c8011 admin admin] PATH_INFO: `/v3/projects/652bc65c53ff4d53b155f229bd0c25e9/users/95b2bf0bbc18466081776c9dda55ac30/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:54.660025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdf9739-b028-4a75-921d-a822251c8011 admin admin] RBAC: Authorizing `identity:create_grant(project_id=652bc65c53ff4d53b155f229bd0c25e9, user_id=95b2bf0bbc18466081776c9dda55ac30, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:54.696335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdf9739-b028-4a75-921d-a822251c8011 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:54.728891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 897/1777] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:13:54 2022] PUT /identity/v3/projects/652bc65c53ff4d53b155f229bd0c25e9/users/95b2bf0bbc18466081776c9dda55ac30/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:13:54.747415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f22c20-931c-4a20-9b47-3873ecff8fa6 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:54.749293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f22c20-931c-4a20-9b47-3873ecff8fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:54.749810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f22c20-931c-4a20-9b47-3873ecff8fa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:54.821227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c8f22c20-931c-4a20-9b47-3873ecff8fa6 None None] MFA Rules not processed for user `95b2bf0bbc18466081776c9dda55ac30`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:54.822753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c8f22c20-931c-4a20-9b47-3873ecff8fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:55.028510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 881/1778] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:54 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:55.045967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdf9739-b028-4a75-921d-a822251c8011 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:55.103317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:55.210176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f22c20-931c-4a20-9b47-3873ecff8fa6 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:55.231928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdf9739-b028-4a75-921d-a822251c8011 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:55.243471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a6eede-9270-4b0e-91b2-f09579d36d94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:55.244391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6eede-9270-4b0e-91b2-f09579d36d94 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:55.244664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6eede-9270-4b0e-91b2-f09579d36d94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:55.244937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6eede-9270-4b0e-91b2-f09579d36d94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:55.262017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a6eede-9270-4b0e-91b2-f09579d36d94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:55.268600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a6eede-9270-4b0e-91b2-f09579d36d94 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:55.292384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 898/1779] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:55 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:55.330577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:55.379268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f22c20-931c-4a20-9b47-3873ecff8fa6 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:55.385379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:55.386320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:55.386697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:55.387079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] PATH_INFO: `/v3/users/525f10462381466397878761517ffcfc` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:55.389073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=525f10462381466397878761517ffcfc)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:55.391971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:55.719599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '525f10462381466397878761517ffcfc'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:55.764774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '525f10462381466397878761517ffcfc'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:55.765797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '525f10462381466397878761517ffcfc'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:55.968852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '525f10462381466397878761517ffcfc'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:55.971160 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553d435-87a2-428d-88f5-4ec49125be43 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:55.978835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553d435-87a2-428d-88f5-4ec49125be43 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:55.978835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553d435-87a2-428d-88f5-4ec49125be43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:55.996802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-084aff09-76e1-4171-bcb3-be0f12fbcb0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '525f10462381466397878761517ffcfc'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:56.015357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 882/1780] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:55 2022] DELETE /identity/v3/users/525f10462381466397878761517ffcfc => generated 0 bytes in 813 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:13:56.021482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7416f2-bcda-4c38-94d4-7527f74c6471 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:56.024135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7416f2-bcda-4c38-94d4-7527f74c6471 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:56.024435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7416f2-bcda-4c38-94d4-7527f74c6471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:56.036351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e553d435-87a2-428d-88f5-4ec49125be43 None None] MFA Rules not processed for user `d61cf1670a7a43519c7258cd467227b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:56.036351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e553d435-87a2-428d-88f5-4ec49125be43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:56.103083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2e7416f2-bcda-4c38-94d4-7527f74c6471 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:56.104667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2e7416f2-bcda-4c38-94d4-7527f74c6471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:56.270847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 899/1781] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:55 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:56.275629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51e84d1-a197-4fe8-8e4e-63124149dcab None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:56.276141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51e84d1-a197-4fe8-8e4e-63124149dcab None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:56.276653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51e84d1-a197-4fe8-8e4e-63124149dcab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:56.320438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a51e84d1-a197-4fe8-8e4e-63124149dcab None None] MFA Rules not processed for user `3326fcf3ccbc4a978e9e52e25456ad98`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:56.324665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a51e84d1-a197-4fe8-8e4e-63124149dcab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:56.444911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 883/1782] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:56.447275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7416f2-bcda-4c38-94d4-7527f74c6471 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:56.461129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2e7416f2-bcda-4c38-94d4-7527f74c6471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:56.472358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 900/1783] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:13:56 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:56.504603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51e84d1-a197-4fe8-8e4e-63124149dcab None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:56.506183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a51e84d1-a197-4fe8-8e4e-63124149dcab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:56.650688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:56.696620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7416f2-bcda-4c38-94d4-7527f74c6471 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:56.720764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:56.727261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:56.727766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:56.731153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:56.734183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51e84d1-a197-4fe8-8e4e-63124149dcab None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:56.737483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:56.742857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff70f7a1-fb33-443c-99c4-f1d61de80ddf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:56.744957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff70f7a1-fb33-443c-99c4-f1d61de80ddf service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:56.746755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff70f7a1-fb33-443c-99c4-f1d61de80ddf service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:56.747050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff70f7a1-fb33-443c-99c4-f1d61de80ddf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:56.787665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff70f7a1-fb33-443c-99c4-f1d61de80ddf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:56.790890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff70f7a1-fb33-443c-99c4-f1d61de80ddf service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:56.831411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 901/1784] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:13:56.832918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664f41b-14f3-4388-902a-a3e9af8421ea None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:56.833205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664f41b-14f3-4388-902a-a3e9af8421ea None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:56.833482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664f41b-14f3-4388-902a-a3e9af8421ea None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:56.834889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 902/1785] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:13:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:13:56.907106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:56.909409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:56.936130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 884/1786] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:56 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:57.026244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbd0c0-d268-40c9-ad37-a796ee8463ea None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:57.029548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbd0c0-d268-40c9-ad37-a796ee8463ea None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:57.029899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbd0c0-d268-40c9-ad37-a796ee8463ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:57.128562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c8bbd0c0-d268-40c9-ad37-a796ee8463ea None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:13:57.132066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c8bbd0c0-d268-40c9-ad37-a796ee8463ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:57.164023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:57.419405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 903/1787] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:13:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 396 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:13:57.433021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:58.594701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:59.141236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:13:59.143369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:59.144855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:59.165884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67408e0e-5a22-4b93-bf20-c821483377f1 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:59.177298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:59.178255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:59.178900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:59.179187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] PATH_INFO: `/v3/projects/dd6717beff0f4843bc9200f907e46dbc` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:59.180405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd6717beff0f4843bc9200f907e46dbc)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:59.182493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:59.341081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd6717beff0f4843bc9200f907e46dbc', 'request_id': 'req-1576a318-664b-48ef-b26f-01b9c5153432'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:59.506791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd6717beff0f4843bc9200f907e46dbc', 'request_id': 'req-1576a318-664b-48ef-b26f-01b9c5153432'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:59.513792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd6717beff0f4843bc9200f907e46dbc', 'request_id': 'req-1576a318-664b-48ef-b26f-01b9c5153432'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:59.527892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bbd0c0-d268-40c9-ad37-a796ee8463ea None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:59.534134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c8bbd0c0-d268-40c9-ad37-a796ee8463ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:59.543231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] The token cache is being invalidate because project dd6717beff0f4843bc9200f907e46dbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:13:59.543782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:59.545883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:13:59.548347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 885/1788] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:13:59 2022] DELETE /identity/v3/projects/dd6717beff0f4843bc9200f907e46dbc => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:13:59.552672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:13:59.558682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bbd0c0-d268-40c9-ad37-a796ee8463ea None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:59.559133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:59.687892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:59.688940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:59.689510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:59.689978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:59.693390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:13:59.699506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1576a318-664b-48ef-b26f-01b9c5153432 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:13:59.706224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:13:59.706948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:13:59.707371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:13:59.707712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] PATH_INFO: `/v3/users/e2fbfb2883ab404eb98a2263b9f018ea` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:13:59.709051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2fbfb2883ab404eb98a2263b9f018ea)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:59.712100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:59.714400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:13:59.716458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:13:59.734871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 904/1789] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:13:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:13:59.985163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2fbfb2883ab404eb98a2263b9f018ea'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:00.000660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2fbfb2883ab404eb98a2263b9f018ea'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:00.001844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2fbfb2883ab404eb98a2263b9f018ea'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:00.135148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2fbfb2883ab404eb98a2263b9f018ea'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:00.151295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0bbc4e18-d10d-41c0-9ea6-b41eb7a095ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2fbfb2883ab404eb98a2263b9f018ea'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:00.158815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 886/1790] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:13:59 2022] DELETE /identity/v3/users/e2fbfb2883ab404eb98a2263b9f018ea => generated 0 bytes in 607 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:00.672947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:01.166065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:01.177249 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4a9850-02e6-4fda-b744-3a47e329d947 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:01.267010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:01.267661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:01.267943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:01.268222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] PATH_INFO: `/v3/projects/98e91b1afc0f4ec89bc81ca41f1ff785` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:01.269447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=98e91b1afc0f4ec89bc81ca41f1ff785)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:01.271503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:01.427884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98e91b1afc0f4ec89bc81ca41f1ff785', 'request_id': 'req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:01.435331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98e91b1afc0f4ec89bc81ca41f1ff785', 'request_id': 'req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:01.641421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98e91b1afc0f4ec89bc81ca41f1ff785', 'request_id': 'req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:01.642186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:01.657284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] The token cache is being invalidate because project 98e91b1afc0f4ec89bc81ca41f1ff785 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:01.657475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:01.658162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0567cdfd-e3e3-4181-91a3-c5c661d52ac9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:01.659662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 905/1791] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:01 2022] DELETE /identity/v3/projects/98e91b1afc0f4ec89bc81ca41f1ff785 => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:01.665086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eaa331-eed4-4351-a959-c6e78c74676c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:01.665435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eaa331-eed4-4351-a959-c6e78c74676c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:01.665931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eaa331-eed4-4351-a959-c6e78c74676c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:01.686874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-97eaa331-eed4-4351-a959-c6e78c74676c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:01.688345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-97eaa331-eed4-4351-a959-c6e78c74676c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:01.757175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 887/1792] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:01.760284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112face1-1c22-4d54-b67f-c40b35bcf25c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:01.762458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112face1-1c22-4d54-b67f-c40b35bcf25c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:01.762961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112face1-1c22-4d54-b67f-c40b35bcf25c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:01.803295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-112face1-1c22-4d54-b67f-c40b35bcf25c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:01.804967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-112face1-1c22-4d54-b67f-c40b35bcf25c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:01.818460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 906/1793] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:01.821271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eaa331-eed4-4351-a959-c6e78c74676c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:01.840984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eaa331-eed4-4351-a959-c6e78c74676c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:01.850493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:01.851144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:01.851390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:01.851761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:01.852458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:01.857967 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:01.860104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:01.875111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 888/1794] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:01.879615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112face1-1c22-4d54-b67f-c40b35bcf25c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:01.890473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112face1-1c22-4d54-b67f-c40b35bcf25c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:01.896227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771e344d-a01d-4905-9cfd-6af4347c639c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:01.896923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771e344d-a01d-4905-9cfd-6af4347c639c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:01.897297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771e344d-a01d-4905-9cfd-6af4347c639c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:01.897666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771e344d-a01d-4905-9cfd-6af4347c639c admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:01.898395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771e344d-a01d-4905-9cfd-6af4347c639c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:01.900728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771e344d-a01d-4905-9cfd-6af4347c639c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:02.005195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 907/1795] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:01 2022] POST /identity/v3/projects => generated 361 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:02.009354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:02.019289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5502ccb-05c8-444b-930b-2ed50b540546 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:02.077079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fb19c6-1096-4fdf-b470-03af6e3c37dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:02.077734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fb19c6-1096-4fdf-b470-03af6e3c37dd admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:02.078018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fb19c6-1096-4fdf-b470-03af6e3c37dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:02.078291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fb19c6-1096-4fdf-b470-03af6e3c37dd admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:02.078932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fb19c6-1096-4fdf-b470-03af6e3c37dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:02.080838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fb19c6-1096-4fdf-b470-03af6e3c37dd admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:02.090273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:02.180550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 889/1796] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:02 2022] POST /identity/v3/users => generated 281 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:02.184685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771e344d-a01d-4905-9cfd-6af4347c639c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:02.196325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771e344d-a01d-4905-9cfd-6af4347c639c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:02.201557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e07f66-8286-4bb5-bed7-a60e29be6520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:02.202396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e07f66-8286-4bb5-bed7-a60e29be6520 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:02.202856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e07f66-8286-4bb5-bed7-a60e29be6520 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:02.203269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e07f66-8286-4bb5-bed7-a60e29be6520 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:02.204061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e07f66-8286-4bb5-bed7-a60e29be6520 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:02.205970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e07f66-8286-4bb5-bed7-a60e29be6520 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:02.230885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 908/1797] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:02 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:02.234498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fb19c6-1096-4fdf-b470-03af6e3c37dd admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:02.245620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fb19c6-1096-4fdf-b470-03af6e3c37dd admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:02.251368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdba384-7bbf-4902-9eef-fb9994b6e008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:02.253847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdba384-7bbf-4902-9eef-fb9994b6e008 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:02.253847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdba384-7bbf-4902-9eef-fb9994b6e008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:02.253847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdba384-7bbf-4902-9eef-fb9994b6e008 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:02.253847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdba384-7bbf-4902-9eef-fb9994b6e008 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:02.255114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdba384-7bbf-4902-9eef-fb9994b6e008 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:02.271738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 890/1798] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:02 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:02.273398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e07f66-8286-4bb5-bed7-a60e29be6520 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:02.284829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e07f66-8286-4bb5-bed7-a60e29be6520 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:02.290297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:02.291012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:02.291355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:02.291695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] PATH_INFO: `/v3/projects/ac3456a509cd44e2853194a3f5292520/users/822861fe1a2c41fc9cabad794c3f2c3e/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:02.320825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac3456a509cd44e2853194a3f5292520, user_id=822861fe1a2c41fc9cabad794c3f2c3e, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:02.322830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:02.409396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 909/1799] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:02 2022] PUT /identity/v3/projects/ac3456a509cd44e2853194a3f5292520/users/822861fe1a2c41fc9cabad794c3f2c3e/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:02.425539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2021207-db10-4b92-8674-ef5d269da1e1 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:02.425860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2021207-db10-4b92-8674-ef5d269da1e1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:02.426139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2021207-db10-4b92-8674-ef5d269da1e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:02.455421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-a2021207-db10-4b92-8674-ef5d269da1e1 None None] MFA Rules not processed for user `822861fe1a2c41fc9cabad794c3f2c3e`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:02.456645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a2021207-db10-4b92-8674-ef5d269da1e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:02.524041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 891/1800] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:02 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:02.568534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:02.570236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:02.625393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc8c03c-566a-4c5f-8734-625a809471e4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:02.629039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30622c31-fc6d-4a80-96ee-69462ff1e6cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:02.629655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30622c31-fc6d-4a80-96ee-69462ff1e6cf service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:02.629996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30622c31-fc6d-4a80-96ee-69462ff1e6cf service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:02.630320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30622c31-fc6d-4a80-96ee-69462ff1e6cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:02.639084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30622c31-fc6d-4a80-96ee-69462ff1e6cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:02.640479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30622c31-fc6d-4a80-96ee-69462ff1e6cf service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:02.649467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 910/1801] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:02 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:02.665185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2021207-db10-4b92-8674-ef5d269da1e1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:02.734104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2021207-db10-4b92-8674-ef5d269da1e1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:02.738757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:02.739461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:02.739806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:02.740090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] PATH_INFO: `/v3/users/822861fe1a2c41fc9cabad794c3f2c3e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:02.741887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] RBAC: Authorizing `identity:delete_user(user_id=822861fe1a2c41fc9cabad794c3f2c3e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:02.744048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:02.931375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '822861fe1a2c41fc9cabad794c3f2c3e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:02.943736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '822861fe1a2c41fc9cabad794c3f2c3e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:02.944417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '822861fe1a2c41fc9cabad794c3f2c3e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:03.026223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '822861fe1a2c41fc9cabad794c3f2c3e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:03.034303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '822861fe1a2c41fc9cabad794c3f2c3e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:03.040268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 892/1802] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:02 2022] DELETE /identity/v3/users/822861fe1a2c41fc9cabad794c3f2c3e => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:03.044871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18329238-cda9-41b5-85f3-6b274a606ae8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:03.045340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18329238-cda9-41b5-85f3-6b274a606ae8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:03.045534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18329238-cda9-41b5-85f3-6b274a606ae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:03.055261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:03.078411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-18329238-cda9-41b5-85f3-6b274a606ae8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:03.079627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-18329238-cda9-41b5-85f3-6b274a606ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:03.140494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 911/1803] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:03.149828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:03.151290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:03.207243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41eb22c-8f35-423d-aabe-7f7911e48943 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:03.210790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7579362f-7bdc-46bb-9ee0-b4fe79ab8619 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:03.211395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7579362f-7bdc-46bb-9ee0-b4fe79ab8619 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:03.211677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7579362f-7bdc-46bb-9ee0-b4fe79ab8619 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:03.211949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7579362f-7bdc-46bb-9ee0-b4fe79ab8619 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:03.213459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:03.221504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7579362f-7bdc-46bb-9ee0-b4fe79ab8619 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:03.222965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7579362f-7bdc-46bb-9ee0-b4fe79ab8619 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:03.233507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 893/1804] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:03.615246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:03.787647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18329238-cda9-41b5-85f3-6b274a606ae8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:03.789143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-18329238-cda9-41b5-85f3-6b274a606ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:03.802178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18329238-cda9-41b5-85f3-6b274a606ae8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:03.807422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:03.808064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:03.808342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:03.812800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] PATH_INFO: `/v3/projects/ac3456a509cd44e2853194a3f5292520` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:03.812800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac3456a509cd44e2853194a3f5292520)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:03.812800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:03.970527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac3456a509cd44e2853194a3f5292520', 'request_id': 'req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:03.975859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac3456a509cd44e2853194a3f5292520', 'request_id': 'req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:04.203615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac3456a509cd44e2853194a3f5292520', 'request_id': 'req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:04.216792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] The token cache is being invalidate because project ac3456a509cd44e2853194a3f5292520 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:04.217087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:04.217794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16cef31c-60e2-4a51-ae6c-37d2ca1ac6c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:04.219705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 912/1805] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:03 2022] DELETE /identity/v3/projects/ac3456a509cd44e2853194a3f5292520 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:04.224465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983c78f-df9f-4b63-a58b-ada82ec558ea None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:04.225379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983c78f-df9f-4b63-a58b-ada82ec558ea None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:04.225379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983c78f-df9f-4b63-a58b-ada82ec558ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:04.247335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-9983c78f-df9f-4b63-a58b-ada82ec558ea None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:04.271392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9983c78f-df9f-4b63-a58b-ada82ec558ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:04.307425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:04.313863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 894/1806] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:04.321480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e415e-06d2-4cd8-b1cc-4b74eeb4ce0e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:04.321803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e415e-06d2-4cd8-b1cc-4b74eeb4ce0e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:04.322079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900e415e-06d2-4cd8-b1cc-4b74eeb4ce0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:04.343038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-900e415e-06d2-4cd8-b1cc-4b74eeb4ce0e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:04.344168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-900e415e-06d2-4cd8-b1cc-4b74eeb4ce0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:04.352130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 913/1807] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:04.356290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9983c78f-df9f-4b63-a58b-ada82ec558ea None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:04.365767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9983c78f-df9f-4b63-a58b-ada82ec558ea None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:04.370523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:04.371129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:04.371408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:04.371681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:04.372865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:04.373169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:04.374798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:04.386310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 895/1808] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:04.391161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900e415e-06d2-4cd8-b1cc-4b74eeb4ce0e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:04.400174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900e415e-06d2-4cd8-b1cc-4b74eeb4ce0e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:04.404941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7209db-a182-41d1-a2bd-521cfa9fc01e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:04.405536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7209db-a182-41d1-a2bd-521cfa9fc01e admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:04.405843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7209db-a182-41d1-a2bd-521cfa9fc01e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:04.406113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7209db-a182-41d1-a2bd-521cfa9fc01e admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:04.406730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7209db-a182-41d1-a2bd-521cfa9fc01e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:04.408605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7209db-a182-41d1-a2bd-521cfa9fc01e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:05.227796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 914/1809] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:04 2022] POST /identity/v3/projects => generated 383 bytes in 837 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:05.236964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:05.246972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c90c8ec-79be-4081-b86e-a0649ebf151f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:05.301307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2960ad1-e566-4626-ac78-0d8fd9f3b067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:05.301874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2960ad1-e566-4626-ac78-0d8fd9f3b067 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:05.302151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2960ad1-e566-4626-ac78-0d8fd9f3b067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:05.302420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2960ad1-e566-4626-ac78-0d8fd9f3b067 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:05.303042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2960ad1-e566-4626-ac78-0d8fd9f3b067 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:05.304952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2960ad1-e566-4626-ac78-0d8fd9f3b067 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:05.435943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 896/1810] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:05 2022] POST /identity/v3/users => generated 292 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:05.437092 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7209db-a182-41d1-a2bd-521cfa9fc01e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:05.447458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7209db-a182-41d1-a2bd-521cfa9fc01e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:05.452535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970986ef-4dd2-4f09-a89f-e60580c2831f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:05.453145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970986ef-4dd2-4f09-a89f-e60580c2831f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:05.453425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970986ef-4dd2-4f09-a89f-e60580c2831f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:05.453732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970986ef-4dd2-4f09-a89f-e60580c2831f admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:05.454454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970986ef-4dd2-4f09-a89f-e60580c2831f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:05.456043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970986ef-4dd2-4f09-a89f-e60580c2831f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:05.466788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 915/1811] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:05.470868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2960ad1-e566-4626-ac78-0d8fd9f3b067 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:05.480860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2960ad1-e566-4626-ac78-0d8fd9f3b067 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:05.486322 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceef077b-2f29-4230-9da9-224fea485660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:05.493748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef077b-2f29-4230-9da9-224fea485660 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:05.493748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef077b-2f29-4230-9da9-224fea485660 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:05.493748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef077b-2f29-4230-9da9-224fea485660 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:05.493748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceef077b-2f29-4230-9da9-224fea485660 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:05.493748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceef077b-2f29-4230-9da9-224fea485660 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:05.500519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 897/1812] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:05.504677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970986ef-4dd2-4f09-a89f-e60580c2831f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:05.516239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970986ef-4dd2-4f09-a89f-e60580c2831f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:05.520771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:05.521417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:05.521731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:05.522008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] PATH_INFO: `/v3/projects/7300ef57c9ac491e98f1537bfcf6cf6e/users/f6cb5f48d04b495cb9bdf544a0990f99/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:05.536744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7300ef57c9ac491e98f1537bfcf6cf6e, user_id=f6cb5f48d04b495cb9bdf544a0990f99, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:05.539511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:05.621283 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 916/1813] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:05 2022] PUT /identity/v3/projects/7300ef57c9ac491e98f1537bfcf6cf6e/users/f6cb5f48d04b495cb9bdf544a0990f99/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:05.631853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268ed02f-139f-4d15-a783-82b16286d7c0 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:05.632027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268ed02f-139f-4d15-a783-82b16286d7c0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:05.632316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268ed02f-139f-4d15-a783-82b16286d7c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:05.660745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-268ed02f-139f-4d15-a783-82b16286d7c0 None None] MFA Rules not processed for user `f6cb5f48d04b495cb9bdf544a0990f99`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:05.661981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-268ed02f-139f-4d15-a783-82b16286d7c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:05.730456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 898/1814] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:05 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:05.742036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:05.743804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:05.813077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ae48df-cb0c-4378-9c17-19ad13be8145 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:05.822423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed1bedf-a590-499e-867d-50a0b87695ee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:05.823117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1bedf-a590-499e-867d-50a0b87695ee service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:05.823401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1bedf-a590-499e-867d-50a0b87695ee service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:05.823673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1bedf-a590-499e-867d-50a0b87695ee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:05.833304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed1bedf-a590-499e-867d-50a0b87695ee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:05.835775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed1bedf-a590-499e-867d-50a0b87695ee service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:05.847538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 917/1815] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:05.951299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:06.231959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:07.413132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:09.003769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:10.167287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:15.173456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6461c1-9e35-4aee-b576-adecb312246e None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:15.174053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6461c1-9e35-4aee-b576-adecb312246e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:15.174053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6461c1-9e35-4aee-b576-adecb312246e None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:15.175377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 899/1816] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:14:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:14:15.186301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5727ef2f-2946-4360-baf4-d491849910dc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:15.186755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5727ef2f-2946-4360-baf4-d491849910dc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:15.186933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5727ef2f-2946-4360-baf4-d491849910dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:15.208345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5727ef2f-2946-4360-baf4-d491849910dc None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:15.209458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5727ef2f-2946-4360-baf4-d491849910dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:15.288650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 918/1817] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:14:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:15.364718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6461c1-9e35-4aee-b576-adecb312246e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:15.453684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6461c1-9e35-4aee-b576-adecb312246e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:15.460003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:15.460590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:15.461082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:15.461396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] PATH_INFO: `/v3/users/f6cb5f48d04b495cb9bdf544a0990f99` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:15.462872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6cb5f48d04b495cb9bdf544a0990f99)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:15.465244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:15.502428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5727ef2f-2946-4360-baf4-d491849910dc None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:15.504067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5727ef2f-2946-4360-baf4-d491849910dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:15.518461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5727ef2f-2946-4360-baf4-d491849910dc None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:15.524665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7bb189-8bac-4758-810f-61f121b01688 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:15.525350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7bb189-8bac-4758-810f-61f121b01688 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:15.525661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7bb189-8bac-4758-810f-61f121b01688 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:15.525932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7bb189-8bac-4758-810f-61f121b01688 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:15.538730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7bb189-8bac-4758-810f-61f121b01688 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:15.550516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7bb189-8bac-4758-810f-61f121b01688 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:15.552587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 919/1818] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:15.780658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:15.786745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6cb5f48d04b495cb9bdf544a0990f99'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:15.799828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6cb5f48d04b495cb9bdf544a0990f99'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:15.800556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6cb5f48d04b495cb9bdf544a0990f99'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:15.945890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6cb5f48d04b495cb9bdf544a0990f99'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:15.954946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6cb5f48d04b495cb9bdf544a0990f99'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:15.961466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 900/1819] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:15 2022] DELETE /identity/v3/users/f6cb5f48d04b495cb9bdf544a0990f99 => generated 0 bytes in 597 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:15.967145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d22fc-54ad-4112-af2a-2e78f258da14 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:15.967429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d22fc-54ad-4112-af2a-2e78f258da14 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:15.967702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d22fc-54ad-4112-af2a-2e78f258da14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:15.987294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8c9d22fc-54ad-4112-af2a-2e78f258da14 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:15.989341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8c9d22fc-54ad-4112-af2a-2e78f258da14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:16.085548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:16.097481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 920/1820] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:16.115593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:16.118274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:16.240347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2625f-9dc0-4033-bc72-06463c90b8c1 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:16.244488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:16.245172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:16.245530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:16.245925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:16.257878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:16.260003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:16.278706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 901/1821] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:16.537192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9d22fc-54ad-4112-af2a-2e78f258da14 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:16.538721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8c9d22fc-54ad-4112-af2a-2e78f258da14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:16.554807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9d22fc-54ad-4112-af2a-2e78f258da14 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:16.560329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:16.561127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:16.561644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:16.562080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] PATH_INFO: `/v3/users/17e369e4e2e3475099521852c727fe59` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:16.563468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=17e369e4e2e3475099521852c727fe59)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:16.565583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:16.781613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:16.783368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:16.796669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0a684c-2908-4ab4-be2d-6cdf0e62c0b8 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:16.802270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:16.807854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:16.807854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:16.807854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] PATH_INFO: `/v3/users/08e3871b74574ad99ce2cf382aa7a69e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:16.807854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=08e3871b74574ad99ce2cf382aa7a69e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:16.807854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:16.927653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17e369e4e2e3475099521852c727fe59'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:16.928780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17e369e4e2e3475099521852c727fe59'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:16.943464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17e369e4e2e3475099521852c727fe59'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.034932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17e369e4e2e3475099521852c727fe59'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.042736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08e3871b74574ad99ce2cf382aa7a69e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.043711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17e369e4e2e3475099521852c727fe59'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.051399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 921/1822] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:16 2022] DELETE /identity/v3/users/17e369e4e2e3475099521852c727fe59 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:17.052350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:17.053708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.061225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08e3871b74574ad99ce2cf382aa7a69e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.062230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08e3871b74574ad99ce2cf382aa7a69e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.118589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da9446b-4009-45a2-9787-edca9cda71b7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:17.123831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:17.124424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.124705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.124982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] PATH_INFO: `/v3/projects/7300ef57c9ac491e98f1537bfcf6cf6e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.126135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7300ef57c9ac491e98f1537bfcf6cf6e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:17.128040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:17.158329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08e3871b74574ad99ce2cf382aa7a69e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.168656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b968fce7-1339-4866-98eb-ab7db67b7d8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08e3871b74574ad99ce2cf382aa7a69e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.175760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 902/1823] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:16 2022] DELETE /identity/v3/users/08e3871b74574ad99ce2cf382aa7a69e => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:17.176881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57382b-4f9b-438f-9913-0e67f49261aa None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.177102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57382b-4f9b-438f-9913-0e67f49261aa None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.177383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57382b-4f9b-438f-9913-0e67f49261aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.202616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5f57382b-4f9b-438f-9913-0e67f49261aa None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:17.203961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5f57382b-4f9b-438f-9913-0e67f49261aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.239184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:17.268746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7300ef57c9ac491e98f1537bfcf6cf6e', 'request_id': 'req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.274565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7300ef57c9ac491e98f1537bfcf6cf6e', 'request_id': 'req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.280207 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 903/1824] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:17.281871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9077df-a154-440d-bdb3-8fb8eec090b6 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.282172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9077df-a154-440d-bdb3-8fb8eec090b6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.282433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9077df-a154-440d-bdb3-8fb8eec090b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.301340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8b9077df-a154-440d-bdb3-8fb8eec090b6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:17.302587 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8b9077df-a154-440d-bdb3-8fb8eec090b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.363920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 904/1825] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:17.366054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9077df-a154-440d-bdb3-8fb8eec090b6 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:17.368667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8b9077df-a154-440d-bdb3-8fb8eec090b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.408144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7300ef57c9ac491e98f1537bfcf6cf6e', 'request_id': 'req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.417497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:17.422150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] The token cache is being invalidate because project 7300ef57c9ac491e98f1537bfcf6cf6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:17.422564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.424902 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:17.430571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 922/1826] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:17 2022] DELETE /identity/v3/projects/7300ef57c9ac491e98f1537bfcf6cf6e => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:17.431838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:17.433484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.451936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9077df-a154-440d-bdb3-8fb8eec090b6 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:17.455509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43227e68-ae84-482d-a3fb-bd66b0135947 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:17.456264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43227e68-ae84-482d-a3fb-bd66b0135947 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.456661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43227e68-ae84-482d-a3fb-bd66b0135947 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.456933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43227e68-ae84-482d-a3fb-bd66b0135947 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.458914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-43227e68-ae84-482d-a3fb-bd66b0135947 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.475260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43227e68-ae84-482d-a3fb-bd66b0135947 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:17.477555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43227e68-ae84-482d-a3fb-bd66b0135947 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:17.488217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 905/1827] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:17.494443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924ffd2e-f3e2-4690-8dce-bce32bc250ae None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.495415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924ffd2e-f3e2-4690-8dce-bce32bc250ae None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.495704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924ffd2e-f3e2-4690-8dce-bce32bc250ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.511441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06743d9-4380-40f9-ae3c-0dbd2f00c64e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:17.515988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:17.516683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.517190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.517549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.518014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-924ffd2e-f3e2-4690-8dce-bce32bc250ae None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:17.519686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.519798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-924ffd2e-f3e2-4690-8dce-bce32bc250ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.529497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 906/1828] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:17.537397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d8b27a-0433-4326-938f-a4f28191bfcd None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.538042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d8b27a-0433-4326-938f-a4f28191bfcd None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.538353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d8b27a-0433-4326-938f-a4f28191bfcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.539104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:17.541283 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:17.555652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 923/1829] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:17.559634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f3d8b27a-0433-4326-938f-a4f28191bfcd None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:17.560874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f3d8b27a-0433-4326-938f-a4f28191bfcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:17.569064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 907/1830] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:17.573208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:17.583799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2cbf-5b23-4aaf-ae03-83bf13c303ad service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:17.588670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:17.590493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.590952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.591242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.591904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:17.592208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:17.593937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:17.607411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 924/1831] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:17.611918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d8b27a-0433-4326-938f-a4f28191bfcd None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:17.621171 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d8b27a-0433-4326-938f-a4f28191bfcd None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:17.627044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73745b41-82b4-4e38-921e-c333d2440719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:17.627651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73745b41-82b4-4e38-921e-c333d2440719 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.627929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73745b41-82b4-4e38-921e-c333d2440719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.628220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73745b41-82b4-4e38-921e-c333d2440719 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.628827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73745b41-82b4-4e38-921e-c333d2440719 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:17.631170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73745b41-82b4-4e38-921e-c333d2440719 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:17.777289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 908/1832] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:17 2022] POST /identity/v3/projects => generated 369 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:17.782236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:17.792674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e57a95e-a40f-451a-b602-68bda134ca37 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:17.893937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db6b963-0ba3-4418-97c5-4bda2b7a7b3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:17.894806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db6b963-0ba3-4418-97c5-4bda2b7a7b3e admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:17.895189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db6b963-0ba3-4418-97c5-4bda2b7a7b3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:17.895897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db6b963-0ba3-4418-97c5-4bda2b7a7b3e admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:17.896619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db6b963-0ba3-4418-97c5-4bda2b7a7b3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:17.899210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db6b963-0ba3-4418-97c5-4bda2b7a7b3e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:18.016700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 925/1833] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:17 2022] POST /identity/v3/users => generated 285 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:18.020899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73745b41-82b4-4e38-921e-c333d2440719 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:18.032273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73745b41-82b4-4e38-921e-c333d2440719 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:18.038317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71ba6ea-2a62-4aee-baa3-f9007d8fda13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:18.038948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71ba6ea-2a62-4aee-baa3-f9007d8fda13 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:18.039230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71ba6ea-2a62-4aee-baa3-f9007d8fda13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:18.039505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71ba6ea-2a62-4aee-baa3-f9007d8fda13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:18.040156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71ba6ea-2a62-4aee-baa3-f9007d8fda13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:18.041922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71ba6ea-2a62-4aee-baa3-f9007d8fda13 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:18.056485 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 909/1834] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:18.057759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db6b963-0ba3-4418-97c5-4bda2b7a7b3e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:18.069571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db6b963-0ba3-4418-97c5-4bda2b7a7b3e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:18.077126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea1bfa5-8861-46a4-b84b-ac869fd5fd71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:18.077947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea1bfa5-8861-46a4-b84b-ac869fd5fd71 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:18.078520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea1bfa5-8861-46a4-b84b-ac869fd5fd71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:18.081053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea1bfa5-8861-46a4-b84b-ac869fd5fd71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:18.081053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea1bfa5-8861-46a4-b84b-ac869fd5fd71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:18.081972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea1bfa5-8861-46a4-b84b-ac869fd5fd71 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:18.097639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 926/1835] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:18 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:18.099700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71ba6ea-2a62-4aee-baa3-f9007d8fda13 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:18.114585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71ba6ea-2a62-4aee-baa3-f9007d8fda13 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:18.122591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:18.123662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:18.124067 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:18.124476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] PATH_INFO: `/v3/projects/dc378b4b85c244699e7ed199fd73d1f4/users/58170b7e613b4f43b7dd7ed8ad27a3fd/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:18.144952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc378b4b85c244699e7ed199fd73d1f4, user_id=58170b7e613b4f43b7dd7ed8ad27a3fd, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:18.148352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:18.251420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 910/1836] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:18 2022] PUT /identity/v3/projects/dc378b4b85c244699e7ed199fd73d1f4/users/58170b7e613b4f43b7dd7ed8ad27a3fd/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:18.261651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf7cd6c-7495-4a54-bb73-a546de8973cf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:18.262857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf7cd6c-7495-4a54-bb73-a546de8973cf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:18.262857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf7cd6c-7495-4a54-bb73-a546de8973cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:18.295096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bdf7cd6c-7495-4a54-bb73-a546de8973cf None None] MFA Rules not processed for user `58170b7e613b4f43b7dd7ed8ad27a3fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:18.297044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bdf7cd6c-7495-4a54-bb73-a546de8973cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:18.381619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 927/1837] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:18.391283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:18.393132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:18.487237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e469b024-6840-4532-b5c7-acd50ac8098f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:18.491644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6cc8f34f9674a94ba23a7157e44a6f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:18.492470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:18.493009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:18.493614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:18.506334 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:18.507802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:18.523499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 911/1838] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:18 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:18.683214 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:18.886436 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf7cd6c-7495-4a54-bb73-a546de8973cf None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:18.890812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bdf7cd6c-7495-4a54-bb73-a546de8973cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:18.894374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:18.895881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:19.035004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf7cd6c-7495-4a54-bb73-a546de8973cf None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:19.042060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e50d00c-506c-4e8d-b7b9-07dcfc4d4c89 service glance] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:19.047113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:19.047769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:19.048046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:19.048322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] PATH_INFO: `/v3/projects/eed65c626fa94427924203ac6546c3be` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:19.051350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] RBAC: Authorizing `identity:delete_project(project_id=eed65c626fa94427924203ac6546c3be)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:19.051773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:19.052427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:19.052714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:19.053012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] PATH_INFO: `/v3/projects/3d75667a7ba740daa9512c0d60782883` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:19.053330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:19.054245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d75667a7ba740daa9512c0d60782883)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:19.056099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:19.234974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eed65c626fa94427924203ac6546c3be', 'request_id': 'req-1ba608c0-4862-451d-a06d-65c4f063eaaf'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.242445 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eed65c626fa94427924203ac6546c3be', 'request_id': 'req-1ba608c0-4862-451d-a06d-65c4f063eaaf'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.270464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d75667a7ba740daa9512c0d60782883', 'request_id': 'req-979cebf0-5c67-4c83-a304-91b0acf18d79'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.304703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:19.341146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eed65c626fa94427924203ac6546c3be', 'request_id': 'req-1ba608c0-4862-451d-a06d-65c4f063eaaf'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.360251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] The token cache is being invalidate because project eed65c626fa94427924203ac6546c3be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:19.360541 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.361219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.363062 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 928/1839] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:18 2022] DELETE /identity/v3/projects/eed65c626fa94427924203ac6546c3be => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:19.366795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:19.368171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:19.454676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d75667a7ba740daa9512c0d60782883', 'request_id': 'req-979cebf0-5c67-4c83-a304-91b0acf18d79'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.470889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d75667a7ba740daa9512c0d60782883', 'request_id': 'req-979cebf0-5c67-4c83-a304-91b0acf18d79'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.481213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba608c0-4862-451d-a06d-65c4f063eaaf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:19.490700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:19.491657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:19.492105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:19.492517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] PATH_INFO: `/v3/users/c9c0d2ce838b4c939bdf1bf15aa39a11` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:19.494291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9c0d2ce838b4c939bdf1bf15aa39a11)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:19.498645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:19.515646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] The token cache is being invalidate because project 3d75667a7ba740daa9512c0d60782883 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:19.516822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.518360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.528442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 912/1840] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:18 2022] DELETE /identity/v3/projects/3d75667a7ba740daa9512c0d60782883 => generated 0 bytes in 631 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:19.529803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:19.535391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:19.680102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979cebf0-5c67-4c83-a304-91b0acf18d79 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:19.686123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:19.687405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:19.687907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:19.688427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] PATH_INFO: `/v3/users/95b2bf0bbc18466081776c9dda55ac30` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:19.690359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=95b2bf0bbc18466081776c9dda55ac30)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:19.692564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:19.865663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9c0d2ce838b4c939bdf1bf15aa39a11'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.868882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9c0d2ce838b4c939bdf1bf15aa39a11'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.894501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9c0d2ce838b4c939bdf1bf15aa39a11'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.958060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95b2bf0bbc18466081776c9dda55ac30'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.980095 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95b2bf0bbc18466081776c9dda55ac30'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.984522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95b2bf0bbc18466081776c9dda55ac30'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:19.993407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9c0d2ce838b4c939bdf1bf15aa39a11'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:20.006684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9c0d2ce838b4c939bdf1bf15aa39a11'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:20.017081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 929/1841] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:19 2022] DELETE /identity/v3/users/c9c0d2ce838b4c939bdf1bf15aa39a11 => generated 0 bytes in 651 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:20.107984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95b2bf0bbc18466081776c9dda55ac30'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:20.119626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95b2bf0bbc18466081776c9dda55ac30'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:20.130812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 913/1842] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:19 2022] DELETE /identity/v3/users/95b2bf0bbc18466081776c9dda55ac30 => generated 0 bytes in 598 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:20.698875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:20.700486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:20.781376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:20.794913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae99485-d96e-47b3-99bb-7b1b628a9da1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:20.861313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:20.862035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:20.862353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:20.862660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] PATH_INFO: `/v3/projects/1d1d7fde888a49b0be635f9dde6002dd` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:20.863994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d1d7fde888a49b0be635f9dde6002dd)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:20.866010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:20.929500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:20.930971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:20.945278 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb5528b-9feb-4f10-877e-99f94b579b3b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:20.953658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:20.954965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:20.955249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:20.955789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] PATH_INFO: `/v3/projects/652bc65c53ff4d53b155f229bd0c25e9` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:20.957145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] RBAC: Authorizing `identity:delete_project(project_id=652bc65c53ff4d53b155f229bd0c25e9)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:20.959386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.011284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d1d7fde888a49b0be635f9dde6002dd', 'request_id': 'req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.019879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d1d7fde888a49b0be635f9dde6002dd', 'request_id': 'req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.054572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:21.101055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '652bc65c53ff4d53b155f229bd0c25e9', 'request_id': 'req-cd251a9e-87bd-4479-9410-96780c70d96d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.117540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d1d7fde888a49b0be635f9dde6002dd', 'request_id': 'req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.131826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] The token cache is being invalidate because project 1d1d7fde888a49b0be635f9dde6002dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:21.132404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.133073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a3872b6a-1df4-4d7a-9ec1-66c34f10f3b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.134688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 930/1843] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:20 2022] DELETE /identity/v3/projects/1d1d7fde888a49b0be635f9dde6002dd => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:21.140028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc10ce4-c15b-4478-a796-35caa0413ea0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.140580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc10ce4-c15b-4478-a796-35caa0413ea0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.140924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc10ce4-c15b-4478-a796-35caa0413ea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.162167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2bc10ce4-c15b-4478-a796-35caa0413ea0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:21.163582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2bc10ce4-c15b-4478-a796-35caa0413ea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:21.237799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 931/1844] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:21.245911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856a5a95-4c51-4dd8-ad8f-3d6bdcf4de0a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.246275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856a5a95-4c51-4dd8-ad8f-3d6bdcf4de0a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.246727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856a5a95-4c51-4dd8-ad8f-3d6bdcf4de0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.269319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-856a5a95-4c51-4dd8-ad8f-3d6bdcf4de0a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:21.271051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-856a5a95-4c51-4dd8-ad8f-3d6bdcf4de0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:21.271374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '652bc65c53ff4d53b155f229bd0c25e9', 'request_id': 'req-cd251a9e-87bd-4479-9410-96780c70d96d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.278738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '652bc65c53ff4d53b155f229bd0c25e9', 'request_id': 'req-cd251a9e-87bd-4479-9410-96780c70d96d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.286900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 932/1845] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:21.290355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] The token cache is being invalidate because project 652bc65c53ff4d53b155f229bd0c25e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:21.290706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.291521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cd251a9e-87bd-4479-9410-96780c70d96d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:21.291883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856a5a95-4c51-4dd8-ad8f-3d6bdcf4de0a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.293008 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 914/1846] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:20 2022] DELETE /identity/v3/projects/652bc65c53ff4d53b155f229bd0c25e9 => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:21.298464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2543d76f-c24b-4080-8310-1cbc0f245ed4 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.298930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2543d76f-c24b-4080-8310-1cbc0f245ed4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.299365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2543d76f-c24b-4080-8310-1cbc0f245ed4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.303176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856a5a95-4c51-4dd8-ad8f-3d6bdcf4de0a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.308860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.309412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.309880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.310152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.310940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:21.311260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.312884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.326116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2543d76f-c24b-4080-8310-1cbc0f245ed4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:21.328144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2543d76f-c24b-4080-8310-1cbc0f245ed4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:21.330733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 933/1847] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:21.337404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.341348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:21.346822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 915/1848] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:21.350199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc4f74f-43e4-4f4b-b394-26e56a28644e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.350492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc4f74f-43e4-4f4b-b394-26e56a28644e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.350769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc4f74f-43e4-4f4b-b394-26e56a28644e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.357998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc2c05-c008-4496-aa9f-8db84c92a725 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.363401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc4260-696d-4c95-b325-38e1e97219b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.364027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc4260-696d-4c95-b325-38e1e97219b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.364316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc4260-696d-4c95-b325-38e1e97219b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.364593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dc4260-696d-4c95-b325-38e1e97219b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.365230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dc4260-696d-4c95-b325-38e1e97219b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.367647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dc4260-696d-4c95-b325-38e1e97219b0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.391142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6fc4f74f-43e4-4f4b-b394-26e56a28644e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:21.392503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6fc4f74f-43e4-4f4b-b394-26e56a28644e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:21.401382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 916/1849] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:21.406113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc4f74f-43e4-4f4b-b394-26e56a28644e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.416079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc4f74f-43e4-4f4b-b394-26e56a28644e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.421009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.421653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.421937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.422203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.422873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:21.423172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.424816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.437191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 917/1850] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:14:21.441402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.452787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0705ba-1901-473c-8450-eb6be5c2c789 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.461230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c410ff-5d64-4536-9ec4-d9c6aa3b97f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.462110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c410ff-5d64-4536-9ec4-d9c6aa3b97f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.462220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c410ff-5d64-4536-9ec4-d9c6aa3b97f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.462476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c410ff-5d64-4536-9ec4-d9c6aa3b97f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.463129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c410ff-5d64-4536-9ec4-d9c6aa3b97f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.465938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c410ff-5d64-4536-9ec4-d9c6aa3b97f8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.474235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 934/1851] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/projects => generated 377 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:21.475415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc4260-696d-4c95-b325-38e1e97219b0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.544821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dc4260-696d-4c95-b325-38e1e97219b0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.549971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3c1a5-7f8a-4458-a85e-f5ef769ed7c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.550573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3c1a5-7f8a-4458-a85e-f5ef769ed7c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.550849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3c1a5-7f8a-4458-a85e-f5ef769ed7c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.551120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3c1a5-7f8a-4458-a85e-f5ef769ed7c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.551822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f3c1a5-7f8a-4458-a85e-f5ef769ed7c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.553766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f3c1a5-7f8a-4458-a85e-f5ef769ed7c9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.576382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 918/1852] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/projects => generated 385 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:21.580095 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c410ff-5d64-4536-9ec4-d9c6aa3b97f8 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.648866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c410ff-5d64-4536-9ec4-d9c6aa3b97f8 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.654872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d942b98a-7141-4544-9225-55d6b54e6ae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.655637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d942b98a-7141-4544-9225-55d6b54e6ae4 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.657287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d942b98a-7141-4544-9225-55d6b54e6ae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.657287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d942b98a-7141-4544-9225-55d6b54e6ae4 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.657287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d942b98a-7141-4544-9225-55d6b54e6ae4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.659347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d942b98a-7141-4544-9225-55d6b54e6ae4 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.660268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 935/1853] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/users => generated 289 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:21.664350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3c1a5-7f8a-4458-a85e-f5ef769ed7c9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.674195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3c1a5-7f8a-4458-a85e-f5ef769ed7c9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.679287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513e34d6-bf6e-461f-a6cd-588cdd5c2014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.680189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513e34d6-bf6e-461f-a6cd-588cdd5c2014 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.680189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513e34d6-bf6e-461f-a6cd-588cdd5c2014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.680448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513e34d6-bf6e-461f-a6cd-588cdd5c2014 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.681121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513e34d6-bf6e-461f-a6cd-588cdd5c2014 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.682756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513e34d6-bf6e-461f-a6cd-588cdd5c2014 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.694618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 936/1854] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:21.698590 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513e34d6-bf6e-461f-a6cd-588cdd5c2014 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.709344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513e34d6-bf6e-461f-a6cd-588cdd5c2014 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.714168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3b288d-c113-4bed-8b02-41529edbdc9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.714739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3b288d-c113-4bed-8b02-41529edbdc9f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.715026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3b288d-c113-4bed-8b02-41529edbdc9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.715415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3b288d-c113-4bed-8b02-41529edbdc9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.716061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3b288d-c113-4bed-8b02-41529edbdc9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.717728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3b288d-c113-4bed-8b02-41529edbdc9f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.729437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 937/1855] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:21.733131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3b288d-c113-4bed-8b02-41529edbdc9f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.743400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3b288d-c113-4bed-8b02-41529edbdc9f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.748514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fada8ad-5175-411d-af79-167d542468a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.749137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fada8ad-5175-411d-af79-167d542468a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.749661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fada8ad-5175-411d-af79-167d542468a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.749849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fada8ad-5175-411d-af79-167d542468a9 admin admin] PATH_INFO: `/v3/projects/f2e5e78515c64a17b6cc18b5270b9d7f/users/640f7e16e79244909c69d0e3581938d0/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.765877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fada8ad-5175-411d-af79-167d542468a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2e5e78515c64a17b6cc18b5270b9d7f, user_id=640f7e16e79244909c69d0e3581938d0, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.768513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fada8ad-5175-411d-af79-167d542468a9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.785330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 919/1856] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/users => generated 293 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:21.789304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d942b98a-7141-4544-9225-55d6b54e6ae4 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.799885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d942b98a-7141-4544-9225-55d6b54e6ae4 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.804584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe4cfc7-8d18-4e06-bdb6-a146e9ef2506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.805201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe4cfc7-8d18-4e06-bdb6-a146e9ef2506 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.805486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe4cfc7-8d18-4e06-bdb6-a146e9ef2506 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.805827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe4cfc7-8d18-4e06-bdb6-a146e9ef2506 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.806476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe4cfc7-8d18-4e06-bdb6-a146e9ef2506 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.808748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe4cfc7-8d18-4e06-bdb6-a146e9ef2506 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.819546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 920/1857] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:21.823434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe4cfc7-8d18-4e06-bdb6-a146e9ef2506 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.834442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe4cfc7-8d18-4e06-bdb6-a146e9ef2506 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.839733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1787664-577f-4e28-a662-fb10ac92126b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.841227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1787664-577f-4e28-a662-fb10ac92126b admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.841645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1787664-577f-4e28-a662-fb10ac92126b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.841993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1787664-577f-4e28-a662-fb10ac92126b admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.842761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1787664-577f-4e28-a662-fb10ac92126b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.844465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1787664-577f-4e28-a662-fb10ac92126b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:21.857645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 921/1858] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:21 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:21.860642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1787664-577f-4e28-a662-fb10ac92126b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:21.867889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 938/1859] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:21 2022] PUT /identity/v3/projects/f2e5e78515c64a17b6cc18b5270b9d7f/users/640f7e16e79244909c69d0e3581938d0/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:21.872506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1787664-577f-4e28-a662-fb10ac92126b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:21.882769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3ce2b-fca8-41f2-a5ca-ce64d956ac7a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.883093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3ce2b-fca8-41f2-a5ca-ce64d956ac7a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.883371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3ce2b-fca8-41f2-a5ca-ce64d956ac7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.920642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-aca3ce2b-fca8-41f2-a5ca-ce64d956ac7a None None] MFA Rules not processed for user `640f7e16e79244909c69d0e3581938d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:21.926531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-aca3ce2b-fca8-41f2-a5ca-ce64d956ac7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:21.948751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45978c9-e79c-4028-9075-00a14fdfe8a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:21.950112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45978c9-e79c-4028-9075-00a14fdfe8a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:21.950455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45978c9-e79c-4028-9075-00a14fdfe8a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:21.950801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45978c9-e79c-4028-9075-00a14fdfe8a3 admin admin] PATH_INFO: `/v3/projects/af2395ace23343bf90d050c03062ed74/users/af960dab84e64f45b52cc820a3f794a2/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:21.969885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45978c9-e79c-4028-9075-00a14fdfe8a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af2395ace23343bf90d050c03062ed74, user_id=af960dab84e64f45b52cc820a3f794a2, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:21.972960 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45978c9-e79c-4028-9075-00a14fdfe8a3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.020247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 939/1860] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:22.024988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca3ce2b-fca8-41f2-a5ca-ce64d956ac7a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.036053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca3ce2b-fca8-41f2-a5ca-ce64d956ac7a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:22.041152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ba5f44-aa95-4411-9c94-47c9de765514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:22.045639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ba5f44-aa95-4411-9c94-47c9de765514 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.045639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ba5f44-aa95-4411-9c94-47c9de765514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.045639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ba5f44-aa95-4411-9c94-47c9de765514 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.045639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ba5f44-aa95-4411-9c94-47c9de765514 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:22.045639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ba5f44-aa95-4411-9c94-47c9de765514 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.061165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 922/1861] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:21 2022] PUT /identity/v3/projects/af2395ace23343bf90d050c03062ed74/users/af960dab84e64f45b52cc820a3f794a2/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:22.071216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84593cc-0edc-4083-a9bb-d8e60452225e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.071499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84593cc-0edc-4083-a9bb-d8e60452225e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.071769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84593cc-0edc-4083-a9bb-d8e60452225e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.101362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-d84593cc-0edc-4083-a9bb-d8e60452225e None None] MFA Rules not processed for user `af960dab84e64f45b52cc820a3f794a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:22.102530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d84593cc-0edc-4083-a9bb-d8e60452225e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:22.158279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 940/1862] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:22 2022] POST /identity/v3/projects => generated 381 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:22.162371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ba5f44-aa95-4411-9c94-47c9de765514 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.173235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 923/1863] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:22.186009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84593cc-0edc-4083-a9bb-d8e60452225e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.187913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d84593cc-0edc-4083-a9bb-d8e60452225e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:22.292812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ba5f44-aa95-4411-9c94-47c9de765514 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:22.296730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84593cc-0edc-4083-a9bb-d8e60452225e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:22.299127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79861627-5c85-4b8e-8d9c-6ba72217a655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:22.299816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79861627-5c85-4b8e-8d9c-6ba72217a655 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.300181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79861627-5c85-4b8e-8d9c-6ba72217a655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.300518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79861627-5c85-4b8e-8d9c-6ba72217a655 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.302363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79861627-5c85-4b8e-8d9c-6ba72217a655 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:22.305370 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79861627-5c85-4b8e-8d9c-6ba72217a655 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.358263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15433591-622a-4708-a6db-c6da421454a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:22.358896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15433591-622a-4708-a6db-c6da421454a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.359176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15433591-622a-4708-a6db-c6da421454a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.365721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15433591-622a-4708-a6db-c6da421454a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.418592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:22.430048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 941/1864] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:22 2022] POST /identity/v3/users => generated 297 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:22.432447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79861627-5c85-4b8e-8d9c-6ba72217a655 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.437990 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15433591-622a-4708-a6db-c6da421454a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:22.439456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15433591-622a-4708-a6db-c6da421454a8 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.450880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 924/1865] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:22 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:22.451667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79861627-5c85-4b8e-8d9c-6ba72217a655 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:22.457033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a41c7d9-bc15-41e3-818b-9f4fac6f2b2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:22.457690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a41c7d9-bc15-41e3-818b-9f4fac6f2b2e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.458006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a41c7d9-bc15-41e3-818b-9f4fac6f2b2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.465694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a41c7d9-bc15-41e3-818b-9f4fac6f2b2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.465694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a41c7d9-bc15-41e3-818b-9f4fac6f2b2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:22.465694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a41c7d9-bc15-41e3-818b-9f4fac6f2b2e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.477735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 942/1866] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:22 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:22.480502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15433591-622a-4708-a6db-c6da421454a8 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.480974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a41c7d9-bc15-41e3-818b-9f4fac6f2b2e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.500231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a41c7d9-bc15-41e3-818b-9f4fac6f2b2e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:22.501328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15433591-622a-4708-a6db-c6da421454a8 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:22.501468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc79e4e-55f7-4491-9c8d-105370d7a281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:22.502181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc79e4e-55f7-4491-9c8d-105370d7a281 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.502576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc79e4e-55f7-4491-9c8d-105370d7a281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.503050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc79e4e-55f7-4491-9c8d-105370d7a281 admin admin] PATH_INFO: `/v3/projects/2e7df7d9bafe4ea8b24ea40e30d75c59/users/5fb976d74df44353a6c83316e50342a5/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.517731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:22.518349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.518634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.518909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] PATH_INFO: `/v3/users/af960dab84e64f45b52cc820a3f794a2` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.523192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] RBAC: Authorizing `identity:delete_user(user_id=af960dab84e64f45b52cc820a3f794a2)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:22.523686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc79e4e-55f7-4491-9c8d-105370d7a281 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e7df7d9bafe4ea8b24ea40e30d75c59, user_id=5fb976d74df44353a6c83316e50342a5, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:22.526464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.534090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc79e4e-55f7-4491-9c8d-105370d7a281 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.665893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 943/1867] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:22 2022] PUT /identity/v3/projects/2e7df7d9bafe4ea8b24ea40e30d75c59/users/5fb976d74df44353a6c83316e50342a5/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:22.676947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc79e4e-55f7-4491-9c8d-105370d7a281 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.716319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:22.767787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc79e4e-55f7-4491-9c8d-105370d7a281 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:22.777736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948eb128-9b9b-4925-9866-402addc61678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:22.778529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948eb128-9b9b-4925-9866-402addc61678 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.778905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948eb128-9b9b-4925-9866-402addc61678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.779300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948eb128-9b9b-4925-9866-402addc61678 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.780052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948eb128-9b9b-4925-9866-402addc61678 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:22.781849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948eb128-9b9b-4925-9866-402addc61678 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.802412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 944/1868] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:22 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:22.807297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948eb128-9b9b-4925-9866-402addc61678 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.819213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948eb128-9b9b-4925-9866-402addc61678 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:22.825165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9239ea-6c34-4918-a497-e08d465bf0de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:22.825945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9239ea-6c34-4918-a497-e08d465bf0de admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.826382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9239ea-6c34-4918-a497-e08d465bf0de admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.826631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9239ea-6c34-4918-a497-e08d465bf0de admin admin] PATH_INFO: `/v3/projects/2e7df7d9bafe4ea8b24ea40e30d75c59/users/5fb976d74df44353a6c83316e50342a5/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.828700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9239ea-6c34-4918-a497-e08d465bf0de admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e7df7d9bafe4ea8b24ea40e30d75c59, user_id=5fb976d74df44353a6c83316e50342a5, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:22.831589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9239ea-6c34-4918-a497-e08d465bf0de admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:22.843734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 945/1869] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:22 2022] PUT /identity/v3/projects/2e7df7d9bafe4ea8b24ea40e30d75c59/users/5fb976d74df44353a6c83316e50342a5/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:22.854093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebe31f-5d5c-4575-a42a-8f3ae66f686c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:22.854385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebe31f-5d5c-4575-a42a-8f3ae66f686c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:22.854671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ebe31f-5d5c-4575-a42a-8f3ae66f686c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:22.876135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af960dab84e64f45b52cc820a3f794a2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:22.889729 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c0ebe31f-5d5c-4575-a42a-8f3ae66f686c None None] MFA Rules not processed for user `5fb976d74df44353a6c83316e50342a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:22.890864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af960dab84e64f45b52cc820a3f794a2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:22.891134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c0ebe31f-5d5c-4575-a42a-8f3ae66f686c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:22.891859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af960dab84e64f45b52cc820a3f794a2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:22.968783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 946/1870] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:22 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:22.982077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ebe31f-5d5c-4575-a42a-8f3ae66f686c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:22.983672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c0ebe31f-5d5c-4575-a42a-8f3ae66f686c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:23.002568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af960dab84e64f45b52cc820a3f794a2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:23.010455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3e4f9d18-950c-4640-871a-f0d7dd42688e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af960dab84e64f45b52cc820a3f794a2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:23.017235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 925/1871] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:22 2022] DELETE /identity/v3/users/af960dab84e64f45b52cc820a3f794a2 => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:23.021640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacead88-ada0-451f-a52c-b7f3d12c73ce None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:23.022330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacead88-ada0-451f-a52c-b7f3d12c73ce None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:23.022456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacead88-ada0-451f-a52c-b7f3d12c73ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:23.046785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-cacead88-ada0-451f-a52c-b7f3d12c73ce None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:23.048119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cacead88-ada0-451f-a52c-b7f3d12c73ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:23.060740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ebe31f-5d5c-4575-a42a-8f3ae66f686c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:23.115302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 926/1872] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:23.120585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:23.121371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:23.121872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:23.122150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:23.125276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacead88-ada0-451f-a52c-b7f3d12c73ce None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:23.127000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cacead88-ada0-451f-a52c-b7f3d12c73ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:23.186543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacead88-ada0-451f-a52c-b7f3d12c73ce None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:23.189087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:23.190433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f869720d-22cb-4d24-925e-a1c28e2574fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:23.190594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:23.191066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869720d-22cb-4d24-925e-a1c28e2574fe service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:23.191346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869720d-22cb-4d24-925e-a1c28e2574fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:23.191628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f869720d-22cb-4d24-925e-a1c28e2574fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:23.200140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 947/1873] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:22 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:23.200727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f869720d-22cb-4d24-925e-a1c28e2574fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:23.202282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f869720d-22cb-4d24-925e-a1c28e2574fe service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:23.211941 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 927/1874] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:23.786382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:23.787896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:23.809543 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298ce474-c9ff-4b74-a4a4-fa327ae5d0a3 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:23.814966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:23.815614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:23.815894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:23.816194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] PATH_INFO: `/v3/projects/af2395ace23343bf90d050c03062ed74` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:23.817362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=af2395ace23343bf90d050c03062ed74)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:23.819383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:23.924077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:23.993233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af2395ace23343bf90d050c03062ed74', 'request_id': 'req-38c47889-a688-4c09-b8b4-eb4114c1fd5f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:23.999451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af2395ace23343bf90d050c03062ed74', 'request_id': 'req-38c47889-a688-4c09-b8b4-eb4114c1fd5f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:24.099918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af2395ace23343bf90d050c03062ed74', 'request_id': 'req-38c47889-a688-4c09-b8b4-eb4114c1fd5f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:24.124411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] The token cache is being invalidate because project af2395ace23343bf90d050c03062ed74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:24.124411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:24.124411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38c47889-a688-4c09-b8b4-eb4114c1fd5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:24.124411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 948/1875] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:23 2022] DELETE /identity/v3/projects/af2395ace23343bf90d050c03062ed74 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:24.126800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4596e0a1-0998-4afc-8737-71b1b091d99b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.127335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4596e0a1-0998-4afc-8737-71b1b091d99b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.127650 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4596e0a1-0998-4afc-8737-71b1b091d99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.150602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4596e0a1-0998-4afc-8737-71b1b091d99b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:24.151830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4596e0a1-0998-4afc-8737-71b1b091d99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:24.237128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 928/1876] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:24.245512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a90ae0-09a8-4b4c-b00c-24f0aed0304a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.245512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a90ae0-09a8-4b4c-b00c-24f0aed0304a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.245512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a90ae0-09a8-4b4c-b00c-24f0aed0304a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.266832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f1a90ae0-09a8-4b4c-b00c-24f0aed0304a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:24.268888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f1a90ae0-09a8-4b4c-b00c-24f0aed0304a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:24.278488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 949/1877] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:24.282566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4596e0a1-0998-4afc-8737-71b1b091d99b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.292342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4596e0a1-0998-4afc-8737-71b1b091d99b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.298132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.299679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:24.299949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.300223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.300499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.302304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:24.302684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.304338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.320079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 929/1878] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:24.324392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a90ae0-09a8-4b4c-b00c-24f0aed0304a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.335112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a90ae0-09a8-4b4c-b00c-24f0aed0304a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.341478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.342130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.345051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.345051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.345051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.345278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.468072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 950/1879] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:24 2022] POST /identity/v3/projects => generated 377 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:24.472349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.490827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf759c4-6aad-4166-a881-68f7e589ee67 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.538134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.540771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:24.568487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989e1dfd-e042-403a-94a2-7d5411707f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.569173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e1dfd-e042-403a-94a2-7d5411707f87 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.569458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e1dfd-e042-403a-94a2-7d5411707f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.569765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e1dfd-e042-403a-94a2-7d5411707f87 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.570437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989e1dfd-e042-403a-94a2-7d5411707f87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.572753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989e1dfd-e042-403a-94a2-7d5411707f87 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.622674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f11455-24cd-4e27-be88-edfe3ee85cc6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.629868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fb976d74df44353a6c83316e50342a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e7df7d9bafe4ea8b24ea40e30d75c59', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.635069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.635069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.635069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.635069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:24.635069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.636147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.656177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 951/1880] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:24.657464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.669530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772057b2-5adc-41d2-9f87-ec3b45088c67 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.675978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c369ad-1314-4e3b-8c53-8dadb57bdbb9 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fb976d74df44353a6c83316e50342a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e7df7d9bafe4ea8b24ea40e30d75c59', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.676907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c369ad-1314-4e3b-8c53-8dadb57bdbb9 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.682891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c369ad-1314-4e3b-8c53-8dadb57bdbb9 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.682891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c369ad-1314-4e3b-8c53-8dadb57bdbb9 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.682891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c369ad-1314-4e3b-8c53-8dadb57bdbb9 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.682891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c369ad-1314-4e3b-8c53-8dadb57bdbb9 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.690847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 930/1881] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:24 2022] POST /identity/v3/users => generated 289 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:24.695699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989e1dfd-e042-403a-94a2-7d5411707f87 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.705873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989e1dfd-e042-403a-94a2-7d5411707f87 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.712136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff456092-9f27-4f0a-8b04-903fdd36edb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.712817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff456092-9f27-4f0a-8b04-903fdd36edb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.713261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff456092-9f27-4f0a-8b04-903fdd36edb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.713528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff456092-9f27-4f0a-8b04-903fdd36edb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.714464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff456092-9f27-4f0a-8b04-903fdd36edb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.716635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff456092-9f27-4f0a-8b04-903fdd36edb9 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.729429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 931/1882] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:24 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:24.734381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff456092-9f27-4f0a-8b04-903fdd36edb9 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.749333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff456092-9f27-4f0a-8b04-903fdd36edb9 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.754792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfed2113-5366-4d1e-85c3-308d58f2b8ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.755499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfed2113-5366-4d1e-85c3-308d58f2b8ea admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.755774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfed2113-5366-4d1e-85c3-308d58f2b8ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.756084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfed2113-5366-4d1e-85c3-308d58f2b8ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.756880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfed2113-5366-4d1e-85c3-308d58f2b8ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.758544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfed2113-5366-4d1e-85c3-308d58f2b8ea admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.768511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 932/1883] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:24 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:24.772615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfed2113-5366-4d1e-85c3-308d58f2b8ea admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.783410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfed2113-5366-4d1e-85c3-308d58f2b8ea admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.791013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50847900-9fb3-4193-87e6-3d1b5fc184ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.791703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50847900-9fb3-4193-87e6-3d1b5fc184ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.791983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50847900-9fb3-4193-87e6-3d1b5fc184ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.792256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50847900-9fb3-4193-87e6-3d1b5fc184ea admin admin] PATH_INFO: `/v3/projects/a6befaa9b9da405dada1e1eda972538e/users/def16806472d43739d61a0b11a5f2ecd/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.802969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 952/1884] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:24 2022] POST /identity/v3/projects => generated 354 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:24.804429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c369ad-1314-4e3b-8c53-8dadb57bdbb9 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.810230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50847900-9fb3-4193-87e6-3d1b5fc184ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6befaa9b9da405dada1e1eda972538e, user_id=def16806472d43739d61a0b11a5f2ecd, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.813170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50847900-9fb3-4193-87e6-3d1b5fc184ea admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.890347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c369ad-1314-4e3b-8c53-8dadb57bdbb9 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.896771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fb976d74df44353a6c83316e50342a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e7df7d9bafe4ea8b24ea40e30d75c59', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:24.897499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.897881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.898164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:24.898838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:24.899191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:24.904028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:24.925893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 953/1885] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:24.965088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:24.979810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 933/1886] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:24 2022] PUT /identity/v3/projects/a6befaa9b9da405dada1e1eda972538e/users/def16806472d43739d61a0b11a5f2ecd/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:24.988266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae17ef97-811e-42d6-8ff6-0781020cb283 tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:24.990590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6322855d-5202-4613-a0ef-289bfd93c446 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:24.991444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6322855d-5202-4613-a0ef-289bfd93c446 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:24.991444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6322855d-5202-4613-a0ef-289bfd93c446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:25.054473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6322855d-5202-4613-a0ef-289bfd93c446 None None] MFA Rules not processed for user `def16806472d43739d61a0b11a5f2ecd`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:25.056281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6322855d-5202-4613-a0ef-289bfd93c446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:25.106242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5fb976d74df44353a6c83316e50342a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2e7df7d9bafe4ea8b24ea40e30d75c59', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:25.106987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:25.107356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:25.107714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] PATH_INFO: `/v3/projects/59ebd184f96249a1920c4c4f4c42ef2d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:25.108963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Authorizing `identity:delete_project(project_id=59ebd184f96249a1920c4c4f4c42ef2d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:25.111234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:25.165038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 934/1887] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:25.166908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6322855d-5202-4613-a0ef-289bfd93c446 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:25.266154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6322855d-5202-4613-a0ef-289bfd93c446 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:25.272095 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862afdf-61a5-40bb-b851-37a3c5ed0a4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:25.272783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862afdf-61a5-40bb-b851-37a3c5ed0a4c admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:25.273997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862afdf-61a5-40bb-b851-37a3c5ed0a4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:25.276339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862afdf-61a5-40bb-b851-37a3c5ed0a4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:25.277150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8862afdf-61a5-40bb-b851-37a3c5ed0a4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:25.279495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8862afdf-61a5-40bb-b851-37a3c5ed0a4c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:25.375691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59ebd184f96249a1920c4c4f4c42ef2d', 'request_id': 'req-7e9ff09b-6184-4435-81e0-b3593fb772ec'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:25.382681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59ebd184f96249a1920c4c4f4c42ef2d', 'request_id': 'req-7e9ff09b-6184-4435-81e0-b3593fb772ec'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:25.475607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 935/1888] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:25 2022] POST /identity/v3/projects => generated 377 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:25.480009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862afdf-61a5-40bb-b851-37a3c5ed0a4c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:25.488721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59ebd184f96249a1920c4c4f4c42ef2d', 'request_id': 'req-7e9ff09b-6184-4435-81e0-b3593fb772ec'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:25.526691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] The token cache is being invalidate because project 59ebd184f96249a1920c4c4f4c42ef2d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:25.527587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:25.530958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:25.533295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 954/1889] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:24 2022] DELETE /identity/v3/projects/59ebd184f96249a1920c4c4f4c42ef2d => generated 0 bytes in 569 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:25.604184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862afdf-61a5-40bb-b851-37a3c5ed0a4c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:25.609513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2af364-28f3-4e84-8236-71f0d8ddec63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:25.610168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2af364-28f3-4e84-8236-71f0d8ddec63 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:25.610448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2af364-28f3-4e84-8236-71f0d8ddec63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:25.610722 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2af364-28f3-4e84-8236-71f0d8ddec63 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:25.611516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2af364-28f3-4e84-8236-71f0d8ddec63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:25.614206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2af364-28f3-4e84-8236-71f0d8ddec63 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:25.734597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 936/1890] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:25 2022] POST /identity/v3/users => generated 289 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:25.738745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:25.740155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:25.756844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:25.821952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9ff09b-6184-4435-81e0-b3593fb772ec tempest-VolumeQuotaClassesTest-2142445610 tempest-VolumeQuotaClassesTest-2142445610-project-admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:25.828376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121d29c-8e4e-4813-940b-5662be20c643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:25.829021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121d29c-8e4e-4813-940b-5662be20c643 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:25.829306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121d29c-8e4e-4813-940b-5662be20c643 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:25.829578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121d29c-8e4e-4813-940b-5662be20c643 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:25.830339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1121d29c-8e4e-4813-940b-5662be20c643 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:25.832027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1121d29c-8e4e-4813-940b-5662be20c643 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:25.850287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 955/1891] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:25.850959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2af364-28f3-4e84-8236-71f0d8ddec63 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:25.864728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2af364-28f3-4e84-8236-71f0d8ddec63 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:25.869786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1461a8-bb21-4d39-8b70-256a9db46645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:25.870505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1461a8-bb21-4d39-8b70-256a9db46645 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:25.870901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1461a8-bb21-4d39-8b70-256a9db46645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:25.871190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1461a8-bb21-4d39-8b70-256a9db46645 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:25.871832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1461a8-bb21-4d39-8b70-256a9db46645 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:25.873538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1461a8-bb21-4d39-8b70-256a9db46645 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:25.888132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 937/1892] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:25.891836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121d29c-8e4e-4813-940b-5662be20c643 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:25.903128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121d29c-8e4e-4813-940b-5662be20c643 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:25.911075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834b969-4f38-408c-bf1b-e813876d2898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:25.912136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834b969-4f38-408c-bf1b-e813876d2898 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:25.917461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834b969-4f38-408c-bf1b-e813876d2898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:25.917461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d834b969-4f38-408c-bf1b-e813876d2898 admin admin] PATH_INFO: `/v3/projects/65d7fe7664b64b0188cde3f1a7b0a031/users/36ea834d1ddf4d029a1c6504b358b15e/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:25.929027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d834b969-4f38-408c-bf1b-e813876d2898 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65d7fe7664b64b0188cde3f1a7b0a031, user_id=36ea834d1ddf4d029a1c6504b358b15e, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:25.931871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d834b969-4f38-408c-bf1b-e813876d2898 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:26.032939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 956/1893] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:25 2022] PUT /identity/v3/projects/65d7fe7664b64b0188cde3f1a7b0a031/users/36ea834d1ddf4d029a1c6504b358b15e/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:26.052284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a8794-c22c-408e-a842-c262a8a3b97e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.052825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a8794-c22c-408e-a842-c262a8a3b97e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.052914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a8794-c22c-408e-a842-c262a8a3b97e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.082024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-cc7a8794-c22c-408e-a842-c262a8a3b97e None None] MFA Rules not processed for user `36ea834d1ddf4d029a1c6504b358b15e`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:26.083226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cc7a8794-c22c-408e-a842-c262a8a3b97e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:26.174148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 938/1894] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:26.178834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834b969-4f38-408c-bf1b-e813876d2898 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:26.298104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d834b969-4f38-408c-bf1b-e813876d2898 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:26.309553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563ec30-c978-4aaa-823a-9058801bcbfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:26.309553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563ec30-c978-4aaa-823a-9058801bcbfc admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.309553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563ec30-c978-4aaa-823a-9058801bcbfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.309553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563ec30-c978-4aaa-823a-9058801bcbfc admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.309862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7563ec30-c978-4aaa-823a-9058801bcbfc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:26.311903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7563ec30-c978-4aaa-823a-9058801bcbfc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:26.322407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7a8794-c22c-408e-a842-c262a8a3b97e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:26.325347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cc7a8794-c22c-408e-a842-c262a8a3b97e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:26.348021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7a8794-c22c-408e-a842-c262a8a3b97e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:26.360680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:26.360680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.360680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.360680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] PATH_INFO: `/v3/users/640f7e16e79244909c69d0e3581938d0` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.361257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=640f7e16e79244909c69d0e3581938d0)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:26.361257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:26.447269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 957/1895] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:26 2022] POST /identity/v3/projects => generated 375 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:26.448546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563ec30-c978-4aaa-823a-9058801bcbfc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:26.516209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563ec30-c978-4aaa-823a-9058801bcbfc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:26.529615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:26.530267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.530546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.530814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.531481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:26.533521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:26.621931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '640f7e16e79244909c69d0e3581938d0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:26.638085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 958/1896] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:26 2022] POST /identity/v3/users => generated 294 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:26.639043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '640f7e16e79244909c69d0e3581938d0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:26.639802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '640f7e16e79244909c69d0e3581938d0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:26.642269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:26.644960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:26.658266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5473ee-685a-4da2-8fa6-62350cb1d709 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:26.663665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba48d1ef-6178-4379-8882-6813af76ce67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:26.664276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48d1ef-6178-4379-8882-6813af76ce67 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.664671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48d1ef-6178-4379-8882-6813af76ce67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.664954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba48d1ef-6178-4379-8882-6813af76ce67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.665624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba48d1ef-6178-4379-8882-6813af76ce67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:26.667203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba48d1ef-6178-4379-8882-6813af76ce67 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:26.683240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 959/1897] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:26 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:26.683348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba48d1ef-6178-4379-8882-6813af76ce67 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:26.693448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba48d1ef-6178-4379-8882-6813af76ce67 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:26.698117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3ab2a4-6e48-496d-aa36-2b34af48e02b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:26.698772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3ab2a4-6e48-496d-aa36-2b34af48e02b admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.699050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3ab2a4-6e48-496d-aa36-2b34af48e02b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.699888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3ab2a4-6e48-496d-aa36-2b34af48e02b admin admin] PATH_INFO: `/v3/projects/f72c99bf3d5e4f688925776aa2952fa1/users/454b999d8bc8404c894d0b587d5d1b19/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.716297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3ab2a4-6e48-496d-aa36-2b34af48e02b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f72c99bf3d5e4f688925776aa2952fa1, user_id=454b999d8bc8404c894d0b587d5d1b19, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:26.719056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3ab2a4-6e48-496d-aa36-2b34af48e02b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:26.763645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '640f7e16e79244909c69d0e3581938d0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:26.776353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-623472f0-52c4-4df9-be0d-4c875add9e74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '640f7e16e79244909c69d0e3581938d0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:26.784930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 939/1898] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:26 2022] DELETE /identity/v3/users/640f7e16e79244909c69d0e3581938d0 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:26.789222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f629857-7971-4358-a63c-c94089386000 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.789608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f629857-7971-4358-a63c-c94089386000 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.789872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f629857-7971-4358-a63c-c94089386000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.812470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 960/1899] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:26 2022] PUT /identity/v3/projects/f72c99bf3d5e4f688925776aa2952fa1/users/454b999d8bc8404c894d0b587d5d1b19/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:26.818945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3ab2a4-6e48-496d-aa36-2b34af48e02b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:26.837651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6f629857-7971-4358-a63c-c94089386000 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:26.842583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6f629857-7971-4358-a63c-c94089386000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:26.945614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3ab2a4-6e48-496d-aa36-2b34af48e02b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:26.951545 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec334367-1bbf-44d9-91b8-6fe2cd79cf20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:26.952915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec334367-1bbf-44d9-91b8-6fe2cd79cf20 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.953547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec334367-1bbf-44d9-91b8-6fe2cd79cf20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.954062 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec334367-1bbf-44d9-91b8-6fe2cd79cf20 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.954504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec334367-1bbf-44d9-91b8-6fe2cd79cf20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:26.956575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec334367-1bbf-44d9-91b8-6fe2cd79cf20 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:26.969015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 961/1900] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:26 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:26.973744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec334367-1bbf-44d9-91b8-6fe2cd79cf20 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:26.985302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec334367-1bbf-44d9-91b8-6fe2cd79cf20 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:26.987085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 940/1901] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:26.994112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50eaf16-741b-4bae-9501-0cb2e2ac7f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:26.995440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50eaf16-741b-4bae-9501-0cb2e2ac7f7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:26.995835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50eaf16-741b-4bae-9501-0cb2e2ac7f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:26.996643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50eaf16-741b-4bae-9501-0cb2e2ac7f7b admin admin] PATH_INFO: `/v3/projects/f72c99bf3d5e4f688925776aa2952fa1/users/454b999d8bc8404c894d0b587d5d1b19/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:26.998896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50eaf16-741b-4bae-9501-0cb2e2ac7f7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f72c99bf3d5e4f688925776aa2952fa1, user_id=454b999d8bc8404c894d0b587d5d1b19, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:26.999906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f629857-7971-4358-a63c-c94089386000 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:27.002036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6f629857-7971-4358-a63c-c94089386000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:27.014337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50eaf16-741b-4bae-9501-0cb2e2ac7f7b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:27.034523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 962/1902] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:26 2022] PUT /identity/v3/projects/f72c99bf3d5e4f688925776aa2952fa1/users/454b999d8bc8404c894d0b587d5d1b19/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:27.039403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130d7019-3d02-4f04-8883-b15ed7ac685f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:27.039706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130d7019-3d02-4f04-8883-b15ed7ac685f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:27.039986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130d7019-3d02-4f04-8883-b15ed7ac685f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:27.075784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-130d7019-3d02-4f04-8883-b15ed7ac685f None None] MFA Rules not processed for user `454b999d8bc8404c894d0b587d5d1b19`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:27.077726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-130d7019-3d02-4f04-8883-b15ed7ac685f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:27.100001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f629857-7971-4358-a63c-c94089386000 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:27.154656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:27.155195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:27.155480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:27.155753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:27.164240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 963/1903] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:27 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:27.175981 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130d7019-3d02-4f04-8883-b15ed7ac685f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:27.177374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-130d7019-3d02-4f04-8883-b15ed7ac685f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:27.210371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:27.244637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:27.246236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:27.253057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130d7019-3d02-4f04-8883-b15ed7ac685f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:27.264229 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 941/1904] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:27.264407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:27.264407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:27.264407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:27.264407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:27.264748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:27.328923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:27.330468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:27.341628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 964/1905] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:27 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:27.606181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:27.994887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:27.997561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:28.054366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da43b0e6-ea6f-44a4-82c8-c6663d8dae2c service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:28.062215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:28.063189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:28.063663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:28.064035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] PATH_INFO: `/v3/projects/f2e5e78515c64a17b6cc18b5270b9d7f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:28.065409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2e5e78515c64a17b6cc18b5270b9d7f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:28.067603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:28.244151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2e5e78515c64a17b6cc18b5270b9d7f', 'request_id': 'req-1ba0d78e-5083-40bf-a197-e85e27952d4d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.332369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2e5e78515c64a17b6cc18b5270b9d7f', 'request_id': 'req-1ba0d78e-5083-40bf-a197-e85e27952d4d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.343646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2e5e78515c64a17b6cc18b5270b9d7f', 'request_id': 'req-1ba0d78e-5083-40bf-a197-e85e27952d4d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.357918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] The token cache is being invalidate because project f2e5e78515c64a17b6cc18b5270b9d7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:28.358402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.360581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.365313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 942/1906] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:27 2022] DELETE /identity/v3/projects/f2e5e78515c64a17b6cc18b5270b9d7f => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:28.366518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:28.381081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e9492c-bd46-4b1c-adc0-7153a94c8cdb service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:28.452848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:28.453516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:28.453838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:28.454112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] PATH_INFO: `/v3/users/5fb976d74df44353a6c83316e50342a5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:28.455247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5fb976d74df44353a6c83316e50342a5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:28.457185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:28.689903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5fb976d74df44353a6c83316e50342a5'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.690575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5fb976d74df44353a6c83316e50342a5'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.702915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5fb976d74df44353a6c83316e50342a5'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.796772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5fb976d74df44353a6c83316e50342a5'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.804483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-16013bc5-b09f-4cd5-ad97-061f4790fc4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5fb976d74df44353a6c83316e50342a5'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:28.810503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 965/1907] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:28 2022] DELETE /identity/v3/users/5fb976d74df44353a6c83316e50342a5 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:28.956814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:29.381119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:29.389736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:29.398388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba0d78e-5083-40bf-a197-e85e27952d4d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:29.460244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:29.461124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:29.461409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:29.461774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] PATH_INFO: `/v3/projects/2e7df7d9bafe4ea8b24ea40e30d75c59` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:29.462945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e7df7d9bafe4ea8b24ea40e30d75c59)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:29.465967 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:29.484369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:29.615693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e7df7d9bafe4ea8b24ea40e30d75c59', 'request_id': 'req-1a043fac-5719-4eb1-b100-202ab353edf3'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:29.705035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e7df7d9bafe4ea8b24ea40e30d75c59', 'request_id': 'req-1a043fac-5719-4eb1-b100-202ab353edf3'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:29.711094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e7df7d9bafe4ea8b24ea40e30d75c59', 'request_id': 'req-1a043fac-5719-4eb1-b100-202ab353edf3'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:29.723113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] The token cache is being invalidate because project 2e7df7d9bafe4ea8b24ea40e30d75c59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:29.723405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:29.724068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a043fac-5719-4eb1-b100-202ab353edf3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:29.725565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 943/1908] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:29 2022] DELETE /identity/v3/projects/2e7df7d9bafe4ea8b24ea40e30d75c59 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:29.731176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bec4df-e2c8-4bcd-9b5a-5b1d38bbe44b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:29.731639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bec4df-e2c8-4bcd-9b5a-5b1d38bbe44b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:29.731725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bec4df-e2c8-4bcd-9b5a-5b1d38bbe44b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:29.752736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f5bec4df-e2c8-4bcd-9b5a-5b1d38bbe44b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:29.754019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f5bec4df-e2c8-4bcd-9b5a-5b1d38bbe44b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:29.815415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 966/1909] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:29.825679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c0ffd-803c-4e4d-ae14-c35565235dcb None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:29.825967 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c0ffd-803c-4e4d-ae14-c35565235dcb None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:29.826242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5c0ffd-803c-4e4d-ae14-c35565235dcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:29.846530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3b5c0ffd-803c-4e4d-ae14-c35565235dcb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:29.847725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3b5c0ffd-803c-4e4d-ae14-c35565235dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:29.856339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 944/1910] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:29.861134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bec4df-e2c8-4bcd-9b5a-5b1d38bbe44b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:29.871067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bec4df-e2c8-4bcd-9b5a-5b1d38bbe44b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:29.873448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:29.877502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:29.885763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:29.885763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:29.885763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:29.885763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:29.885763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:29.885763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:29.895453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 967/1911] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:29.899938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5c0ffd-803c-4e4d-ae14-c35565235dcb None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:29.915049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5c0ffd-803c-4e4d-ae14-c35565235dcb None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:29.916815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c7ec20-1462-4a08-897c-d47f8a5fbb3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:29.917432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c7ec20-1462-4a08-897c-d47f8a5fbb3c admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:29.917772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c7ec20-1462-4a08-897c-d47f8a5fbb3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:29.918673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c7ec20-1462-4a08-897c-d47f8a5fbb3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:29.919303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c7ec20-1462-4a08-897c-d47f8a5fbb3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:29.921305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c7ec20-1462-4a08-897c-d47f8a5fbb3c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:30.067399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 945/1912] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:29 2022] POST /identity/v3/projects => generated 401 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:30.072692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:30.082627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523fd1b1-9ebe-4d9f-ae7e-1a55748892ec admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:30.141671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df4735-3f63-4196-a1d8-586d3c940777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:30.142322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df4735-3f63-4196-a1d8-586d3c940777 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:30.142987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df4735-3f63-4196-a1d8-586d3c940777 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:30.142987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df4735-3f63-4196-a1d8-586d3c940777 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:30.143452 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35df4735-3f63-4196-a1d8-586d3c940777 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:30.146625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35df4735-3f63-4196-a1d8-586d3c940777 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:30.266133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 968/1913] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:30 2022] POST /identity/v3/users => generated 301 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:30.273854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c7ec20-1462-4a08-897c-d47f8a5fbb3c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:30.286209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c7ec20-1462-4a08-897c-d47f8a5fbb3c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:30.290944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5949f383-f348-4cfa-8224-683ff3632e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:30.291553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949f383-f348-4cfa-8224-683ff3632e67 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:30.291837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949f383-f348-4cfa-8224-683ff3632e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:30.292110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5949f383-f348-4cfa-8224-683ff3632e67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:30.292749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5949f383-f348-4cfa-8224-683ff3632e67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:30.294408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5949f383-f348-4cfa-8224-683ff3632e67 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:30.304239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 946/1914] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:30.308108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df4735-3f63-4196-a1d8-586d3c940777 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:30.317947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df4735-3f63-4196-a1d8-586d3c940777 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:30.322678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb21b23-5d24-44d1-ae12-f537a0ccb2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:30.323357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb21b23-5d24-44d1-ae12-f537a0ccb2d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:30.323641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb21b23-5d24-44d1-ae12-f537a0ccb2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:30.323914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb21b23-5d24-44d1-ae12-f537a0ccb2d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:30.325077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb21b23-5d24-44d1-ae12-f537a0ccb2d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:30.326694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb21b23-5d24-44d1-ae12-f537a0ccb2d4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:30.342598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 969/1915] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:30.343832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5949f383-f348-4cfa-8224-683ff3632e67 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:30.354279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5949f383-f348-4cfa-8224-683ff3632e67 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:30.359141 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e484c1b0-7bb7-4adf-b0a8-c083a55ec757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:30.360042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e484c1b0-7bb7-4adf-b0a8-c083a55ec757 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:30.360323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e484c1b0-7bb7-4adf-b0a8-c083a55ec757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:30.360604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e484c1b0-7bb7-4adf-b0a8-c083a55ec757 admin admin] PATH_INFO: `/v3/projects/8f02c20845a24ccda238042339dce328/users/f916a4b2812d41c4bfc92f42167c7e16/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:30.376470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e484c1b0-7bb7-4adf-b0a8-c083a55ec757 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f02c20845a24ccda238042339dce328, user_id=f916a4b2812d41c4bfc92f42167c7e16, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:30.379900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e484c1b0-7bb7-4adf-b0a8-c083a55ec757 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:30.562594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 947/1916] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:30 2022] PUT /identity/v3/projects/8f02c20845a24ccda238042339dce328/users/f916a4b2812d41c4bfc92f42167c7e16/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:30.572923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8063e6-5017-4ce8-80d0-b144d5af506b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:30.573208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8063e6-5017-4ce8-80d0-b144d5af506b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:30.573502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8063e6-5017-4ce8-80d0-b144d5af506b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:30.592024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:30.606806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0a8063e6-5017-4ce8-80d0-b144d5af506b None None] MFA Rules not processed for user `f916a4b2812d41c4bfc92f42167c7e16`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:30.608031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0a8063e6-5017-4ce8-80d0-b144d5af506b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:30.676805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 970/1917] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:30 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:30.681241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e484c1b0-7bb7-4adf-b0a8-c083a55ec757 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:30.724689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8063e6-5017-4ce8-80d0-b144d5af506b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:30.726145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0a8063e6-5017-4ce8-80d0-b144d5af506b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:30.770138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e484c1b0-7bb7-4adf-b0a8-c083a55ec757 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:30.775889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17670239-d6c7-4ea6-bc36-5372430ba35d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:30.778306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17670239-d6c7-4ea6-bc36-5372430ba35d admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:30.779670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17670239-d6c7-4ea6-bc36-5372430ba35d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:30.779670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17670239-d6c7-4ea6-bc36-5372430ba35d admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:30.779670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17670239-d6c7-4ea6-bc36-5372430ba35d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:30.781560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17670239-d6c7-4ea6-bc36-5372430ba35d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:30.790671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8063e6-5017-4ce8-80d0-b144d5af506b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:30.794400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:30.795021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:30.795467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:30.795998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:30.798476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:30.855788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:30.857187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:30.858945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 948/1918] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:30 2022] POST /identity/v3/projects => generated 401 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:30.863010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17670239-d6c7-4ea6-bc36-5372430ba35d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:30.866175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 971/1919] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:30 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:30.928042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17670239-d6c7-4ea6-bc36-5372430ba35d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:30.932654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f03c6ff-14d7-4ec9-a33f-c5c9d84b3950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:30.933264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f03c6ff-14d7-4ec9-a33f-c5c9d84b3950 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:30.934302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f03c6ff-14d7-4ec9-a33f-c5c9d84b3950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:30.934432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f03c6ff-14d7-4ec9-a33f-c5c9d84b3950 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:30.935057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f03c6ff-14d7-4ec9-a33f-c5c9d84b3950 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:30.939711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f03c6ff-14d7-4ec9-a33f-c5c9d84b3950 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:31.013837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 949/1920] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:30 2022] POST /identity/v3/users => generated 307 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:31.017856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:31.028349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a28f8ea-5ede-4077-ac07-9c4177ff9101 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:31.032919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a857cb-5488-4e32-8c9e-52f0bd72ce55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:31.033512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a857cb-5488-4e32-8c9e-52f0bd72ce55 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:31.033824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a857cb-5488-4e32-8c9e-52f0bd72ce55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:31.034098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a857cb-5488-4e32-8c9e-52f0bd72ce55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:31.034770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a857cb-5488-4e32-8c9e-52f0bd72ce55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:31.036358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a857cb-5488-4e32-8c9e-52f0bd72ce55 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:31.046581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 972/1921] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:31.053016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f03c6ff-14d7-4ec9-a33f-c5c9d84b3950 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:31.063489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f03c6ff-14d7-4ec9-a33f-c5c9d84b3950 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:31.069557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152d627b-60ad-4404-b0e9-1731b8d6838a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:31.069557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d627b-60ad-4404-b0e9-1731b8d6838a admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:31.069557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d627b-60ad-4404-b0e9-1731b8d6838a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:31.069557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d627b-60ad-4404-b0e9-1731b8d6838a admin admin] PATH_INFO: `/v3/projects/44c42c9bd9a6469b8dd8885d15def246/users/4f9beef772ba47509a0195f354a75300/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:31.084272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152d627b-60ad-4404-b0e9-1731b8d6838a admin admin] RBAC: Authorizing `identity:create_grant(project_id=44c42c9bd9a6469b8dd8885d15def246, user_id=4f9beef772ba47509a0195f354a75300, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:31.086969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152d627b-60ad-4404-b0e9-1731b8d6838a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:31.144140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:31.222778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 950/1922] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:31 2022] PUT /identity/v3/projects/44c42c9bd9a6469b8dd8885d15def246/users/4f9beef772ba47509a0195f354a75300/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:31.226838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a857cb-5488-4e32-8c9e-52f0bd72ce55 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:31.235973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:31.240947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a857cb-5488-4e32-8c9e-52f0bd72ce55 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:31.317309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f6ceed-f53d-45ba-b13c-8ba12ce679af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:31.317962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6ceed-f53d-45ba-b13c-8ba12ce679af admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:31.318264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6ceed-f53d-45ba-b13c-8ba12ce679af admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:31.318537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6ceed-f53d-45ba-b13c-8ba12ce679af admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:31.319294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f6ceed-f53d-45ba-b13c-8ba12ce679af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:31.320947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f6ceed-f53d-45ba-b13c-8ba12ce679af admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:31.332748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 973/1923] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:31 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:31.342752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152d627b-60ad-4404-b0e9-1731b8d6838a admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:31.353892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152d627b-60ad-4404-b0e9-1731b8d6838a admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:31.359525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:31.360190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:31.360468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:31.361094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] PATH_INFO: `/v3/projects/44c42c9bd9a6469b8dd8885d15def246/users/4f9beef772ba47509a0195f354a75300/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:31.364148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=44c42c9bd9a6469b8dd8885d15def246, user_id=4f9beef772ba47509a0195f354a75300, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:31.365794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:31.378481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 951/1924] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:31 2022] PUT /identity/v3/projects/44c42c9bd9a6469b8dd8885d15def246/users/4f9beef772ba47509a0195f354a75300/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:31.389218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c967d-d8c0-4113-ac34-4bb9fde8cbf0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:31.389500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c967d-d8c0-4113-ac34-4bb9fde8cbf0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:31.393065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0c967d-d8c0-4113-ac34-4bb9fde8cbf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:31.425745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8b0c967d-d8c0-4113-ac34-4bb9fde8cbf0 None None] MFA Rules not processed for user `4f9beef772ba47509a0195f354a75300`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:31.427185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8b0c967d-d8c0-4113-ac34-4bb9fde8cbf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:31.496864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 974/1925] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:31 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:31.508064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:31.509792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:31.573788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bfb92d-d4ac-446d-a5b3-951f3a831c5c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:31.578022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b605c5-81fd-408b-900d-83d3620bd3e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:31.578649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b605c5-81fd-408b-900d-83d3620bd3e0 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:31.578933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b605c5-81fd-408b-900d-83d3620bd3e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:31.579208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b605c5-81fd-408b-900d-83d3620bd3e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:31.588402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b605c5-81fd-408b-900d-83d3620bd3e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:31.590218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b605c5-81fd-408b-900d-83d3620bd3e0 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:31.599846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 952/1926] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:31 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:31.789526 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:31.881116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0c967d-d8c0-4113-ac34-4bb9fde8cbf0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:31.892921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0c967d-d8c0-4113-ac34-4bb9fde8cbf0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:31.896450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6999243b-b952-4b57-8968-65cc3818e820 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:31.897060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6999243b-b952-4b57-8968-65cc3818e820 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:31.897334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6999243b-b952-4b57-8968-65cc3818e820 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:31.897700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6999243b-b952-4b57-8968-65cc3818e820 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:31.939558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:31.957144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6999243b-b952-4b57-8968-65cc3818e820 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:31.958638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6999243b-b952-4b57-8968-65cc3818e820 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:31.968147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 975/1927] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:31 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:32.891065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b605c5-81fd-408b-900d-83d3620bd3e0 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:32.901437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b605c5-81fd-408b-900d-83d3620bd3e0 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:32.906185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:32.906872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:32.909860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:32.909860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:32.909860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:32.972401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:32.973133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:32.974594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:32.984705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 953/1928] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:32 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:33.390056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:33.792448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:34.276436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:34.349918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:34.972726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:40.120314 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6999243b-b952-4b57-8968-65cc3818e820 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:40.188651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6999243b-b952-4b57-8968-65cc3818e820 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:40.195064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:40.196234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:40.196234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:40.196446 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] PATH_INFO: `/v3/users/f916a4b2812d41c4bfc92f42167c7e16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:40.197661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] RBAC: Authorizing `identity:delete_user(user_id=f916a4b2812d41c4bfc92f42167c7e16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:40.199707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:40.299959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:40.340828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:40.342871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:40.356807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8822166-294f-42bf-816e-ca48509f853d service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:40.361644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:40.362428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:40.362715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:40.363281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] PATH_INFO: `/v3/users/def16806472d43739d61a0b11a5f2ecd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:40.364520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] RBAC: Authorizing `identity:delete_user(user_id=def16806472d43739d61a0b11a5f2ecd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:40.366630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:40.461036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f916a4b2812d41c4bfc92f42167c7e16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.461717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f916a4b2812d41c4bfc92f42167c7e16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.474626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f916a4b2812d41c4bfc92f42167c7e16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.608151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f916a4b2812d41c4bfc92f42167c7e16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.616668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c2a3bcf0-a610-4315-afba-e0c2af24712c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f916a4b2812d41c4bfc92f42167c7e16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.622657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'def16806472d43739d61a0b11a5f2ecd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.623854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 976/1929] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:40 2022] DELETE /identity/v3/users/f916a4b2812d41c4bfc92f42167c7e16 => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:40.628308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea59e4bf-89ea-44ff-85c8-ffd3d7b75c3c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:40.628689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea59e4bf-89ea-44ff-85c8-ffd3d7b75c3c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:40.629966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea59e4bf-89ea-44ff-85c8-ffd3d7b75c3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:40.636180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'def16806472d43739d61a0b11a5f2ecd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.636998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'def16806472d43739d61a0b11a5f2ecd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.670530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ea59e4bf-89ea-44ff-85c8-ffd3d7b75c3c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:40.674495 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ea59e4bf-89ea-44ff-85c8-ffd3d7b75c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:40.760498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 977/1930] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:40.778663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea59e4bf-89ea-44ff-85c8-ffd3d7b75c3c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:40.781461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ea59e4bf-89ea-44ff-85c8-ffd3d7b75c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:40.795538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'def16806472d43739d61a0b11a5f2ecd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.812075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1a0e4fc5-2828-4e50-ab5a-df742421985f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'def16806472d43739d61a0b11a5f2ecd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:40.821727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 954/1931] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:40 2022] DELETE /identity/v3/users/def16806472d43739d61a0b11a5f2ecd => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:40.826649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d874c618-3eed-4060-9646-036ddfccd70f None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:40.827673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d874c618-3eed-4060-9646-036ddfccd70f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:40.828132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d874c618-3eed-4060-9646-036ddfccd70f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:40.863504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-d874c618-3eed-4060-9646-036ddfccd70f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:40.863504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d874c618-3eed-4060-9646-036ddfccd70f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:40.883488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea59e4bf-89ea-44ff-85c8-ffd3d7b75c3c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:40.945283 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:40.972016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:40.972016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:40.972016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:40.972501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 955/1932] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:40.972691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:40.972774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d874c618-3eed-4060-9646-036ddfccd70f None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:40.972774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d874c618-3eed-4060-9646-036ddfccd70f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:40.984685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d874c618-3eed-4060-9646-036ddfccd70f None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:40.989778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:40.990679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:40.990848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:40.991579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:41.003987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:41.005988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:41.026091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:41.028425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:41.032362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 956/1933] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:41.041349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 978/1934] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:41.693841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:41.694494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:41.694876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:41.694997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:41.719583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2329f2c0-5f9f-4606-960f-1b87ca13242f service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:41.728599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:41.728931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c8595-aabd-4358-a829-bfdef01b7df3 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:41.729027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:41.729027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:41.729027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] PATH_INFO: `/v3/projects/8f02c20845a24ccda238042339dce328` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:41.729027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f02c20845a24ccda238042339dce328)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:41.730625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:41.731272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:41.731682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:41.731862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] PATH_INFO: `/v3/projects/a6befaa9b9da405dada1e1eda972538e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:41.734627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:41.737763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6befaa9b9da405dada1e1eda972538e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:41.739705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:41.902129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:41.904493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f02c20845a24ccda238042339dce328', 'request_id': 'req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:41.905050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6befaa9b9da405dada1e1eda972538e', 'request_id': 'req-1702786b-8fa2-422d-8db1-ae53133da693'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:41.913375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6befaa9b9da405dada1e1eda972538e', 'request_id': 'req-1702786b-8fa2-422d-8db1-ae53133da693'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.001665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f02c20845a24ccda238042339dce328', 'request_id': 'req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.007306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f02c20845a24ccda238042339dce328', 'request_id': 'req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.021127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] The token cache is being invalidate because project 8f02c20845a24ccda238042339dce328 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:42.021422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.022119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.024328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 957/1935] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:41 2022] DELETE /identity/v3/projects/8f02c20845a24ccda238042339dce328 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:42.028121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:42.030645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:42.107072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca5517c-88fa-4e54-a9b1-d1018ea30a65 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:42.116667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6befaa9b9da405dada1e1eda972538e', 'request_id': 'req-1702786b-8fa2-422d-8db1-ae53133da693'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.130052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] The token cache is being invalidate because project a6befaa9b9da405dada1e1eda972538e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:42.130425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.131298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.135699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 979/1936] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:41 2022] DELETE /identity/v3/projects/a6befaa9b9da405dada1e1eda972538e => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:42.137232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:42.138630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:42.175159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:42.175889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:42.176339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:42.176657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] PATH_INFO: `/v3/users/4f9beef772ba47509a0195f354a75300` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:42.177977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f9beef772ba47509a0195f354a75300)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:42.180063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:42.222477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1702786b-8fa2-422d-8db1-ae53133da693 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:42.227906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:42.228557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:42.228841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:42.229176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] PATH_INFO: `/v3/users/36ea834d1ddf4d029a1c6504b358b15e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:42.233451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=36ea834d1ddf4d029a1c6504b358b15e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:42.235476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:42.408280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f9beef772ba47509a0195f354a75300'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.421387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f9beef772ba47509a0195f354a75300'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.422146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f9beef772ba47509a0195f354a75300'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.488961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:42.494647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36ea834d1ddf4d029a1c6504b358b15e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.495329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36ea834d1ddf4d029a1c6504b358b15e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.505950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f9beef772ba47509a0195f354a75300'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.514179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36ea834d1ddf4d029a1c6504b358b15e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.516600 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f9beef772ba47509a0195f354a75300'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.528070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 958/1937] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:42 2022] DELETE /identity/v3/users/4f9beef772ba47509a0195f354a75300 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:42.579796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36ea834d1ddf4d029a1c6504b358b15e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.594086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36ea834d1ddf4d029a1c6504b358b15e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:42.608337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 980/1938] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:42 2022] DELETE /identity/v3/users/36ea834d1ddf4d029a1c6504b358b15e => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:43.159304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:43.160865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:43.166691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:43.200466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:43.202260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:43.283776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2008d-f9b0-4a3e-abfe-593ba2b3c995 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:43.294724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:43.298704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:43.298704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:43.298704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] PATH_INFO: `/v3/projects/44c42c9bd9a6469b8dd8885d15def246` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:43.298704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] RBAC: Authorizing `identity:delete_project(project_id=44c42c9bd9a6469b8dd8885d15def246)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:43.299576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:43.301458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913e5437-4071-45b4-8c9a-259685133c0b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:43.306663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:43.307067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:43.307362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:43.307635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] PATH_INFO: `/v3/projects/65d7fe7664b64b0188cde3f1a7b0a031` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:43.308873 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65d7fe7664b64b0188cde3f1a7b0a031)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:43.310856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:43.504896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44c42c9bd9a6469b8dd8885d15def246', 'request_id': 'req-d7794871-e253-41e8-b9da-ccbd16124a4d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.551504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65d7fe7664b64b0188cde3f1a7b0a031', 'request_id': 'req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.556925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65d7fe7664b64b0188cde3f1a7b0a031', 'request_id': 'req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.625187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44c42c9bd9a6469b8dd8885d15def246', 'request_id': 'req-d7794871-e253-41e8-b9da-ccbd16124a4d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.631231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44c42c9bd9a6469b8dd8885d15def246', 'request_id': 'req-d7794871-e253-41e8-b9da-ccbd16124a4d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.642434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] The token cache is being invalidate because project 44c42c9bd9a6469b8dd8885d15def246 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:43.642724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.643364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d7794871-e253-41e8-b9da-ccbd16124a4d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.644853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 959/1939] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:43 2022] DELETE /identity/v3/projects/44c42c9bd9a6469b8dd8885d15def246 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:43.649515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bffe6a0-5da3-43c7-9518-503b9d25ca8d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:43.649835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bffe6a0-5da3-43c7-9518-503b9d25ca8d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:43.650115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bffe6a0-5da3-43c7-9518-503b9d25ca8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:43.671746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6bffe6a0-5da3-43c7-9518-503b9d25ca8d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:43.672957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6bffe6a0-5da3-43c7-9518-503b9d25ca8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:43.726211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65d7fe7664b64b0188cde3f1a7b0a031', 'request_id': 'req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.734853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 960/1940] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:43.746691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] The token cache is being invalidate because project 65d7fe7664b64b0188cde3f1a7b0a031 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:43.747169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d56149-cac7-4414-ac5c-2e0d7812d93a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:43.747328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.748500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d56149-cac7-4414-ac5c-2e0d7812d93a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:43.748821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:43.749247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d56149-cac7-4414-ac5c-2e0d7812d93a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:43.752414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 981/1941] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:43 2022] DELETE /identity/v3/projects/65d7fe7664b64b0188cde3f1a7b0a031 => generated 0 bytes in 552 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:43.758488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:43.762550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:43.779044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-00d56149-cac7-4414-ac5c-2e0d7812d93a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:43.780325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-00d56149-cac7-4414-ac5c-2e0d7812d93a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:43.787217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf49998b-98e2-4a5b-a2f9-2ed1e0176094 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:43.789636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 961/1942] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:43.793072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d56149-cac7-4414-ac5c-2e0d7812d93a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:43.802783 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d56149-cac7-4414-ac5c-2e0d7812d93a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:43.806162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:43.807522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:43.807801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:43.808793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] PATH_INFO: `/v3/users/454b999d8bc8404c894d0b587d5d1b19` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:43.810106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] RBAC: Authorizing `identity:delete_user(user_id=454b999d8bc8404c894d0b587d5d1b19)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:43.812523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:43.812654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:43.813475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:43.813798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:43.816005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:43.817178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:43.817503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:43.820956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:43.837767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 962/1943] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:14:43.842564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:43.866206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055f46c6-9c58-4c00-a8e9-2d1853af2fcc admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:43.871376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04967e10-e4ec-43fa-a96b-36d2770eb79a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:43.872217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04967e10-e4ec-43fa-a96b-36d2770eb79a admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:43.872582 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04967e10-e4ec-43fa-a96b-36d2770eb79a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:43.872922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04967e10-e4ec-43fa-a96b-36d2770eb79a admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:43.873736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04967e10-e4ec-43fa-a96b-36d2770eb79a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:43.875851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04967e10-e4ec-43fa-a96b-36d2770eb79a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:43.888635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:44.026424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 963/1944] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:43 2022] POST /identity/v3/projects => generated 387 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:44.031378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04967e10-e4ec-43fa-a96b-36d2770eb79a admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:44.106044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04967e10-e4ec-43fa-a96b-36d2770eb79a admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:44.109578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:44.111756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:44.112405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:44.112691 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:44.112971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:44.113630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:44.115709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:44.115835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '454b999d8bc8404c894d0b587d5d1b19'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:44.116604 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '454b999d8bc8404c894d0b587d5d1b19'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:44.146912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '454b999d8bc8404c894d0b587d5d1b19'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:44.259543 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '454b999d8bc8404c894d0b587d5d1b19'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:44.270601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '454b999d8bc8404c894d0b587d5d1b19'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:44.275190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 964/1945] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:44 2022] POST /identity/v3/users => generated 294 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:44.279598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:44.283505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 982/1946] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:43 2022] DELETE /identity/v3/users/454b999d8bc8404c894d0b587d5d1b19 => generated 0 bytes in 526 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:44.287124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:44.306185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd8dc1-2224-4f5a-8a57-d07102dcede0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:44.385676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75c9d57-f5cc-40f6-9934-feb296bee3b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:44.386316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75c9d57-f5cc-40f6-9934-feb296bee3b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:44.386616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75c9d57-f5cc-40f6-9934-feb296bee3b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:44.386891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75c9d57-f5cc-40f6-9934-feb296bee3b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:44.387588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75c9d57-f5cc-40f6-9934-feb296bee3b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:44.389195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75c9d57-f5cc-40f6-9934-feb296bee3b0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:44.400628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 965/1947] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:44 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:44.405247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:44.416266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db26ba2c-5626-4d60-9dd5-dcdf09487634 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:44.423086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171d078f-d493-44c5-90f8-931e92232322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:44.423874 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171d078f-d493-44c5-90f8-931e92232322 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:44.424226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171d078f-d493-44c5-90f8-931e92232322 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:44.424585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171d078f-d493-44c5-90f8-931e92232322 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:44.425921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171d078f-d493-44c5-90f8-931e92232322 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:44.427613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171d078f-d493-44c5-90f8-931e92232322 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:44.440319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 983/1948] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:44 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:44.450181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75c9d57-f5cc-40f6-9934-feb296bee3b0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:44.460655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75c9d57-f5cc-40f6-9934-feb296bee3b0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:44.466746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aa6783-86f5-457e-af98-633586d322f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:44.467422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa6783-86f5-457e-af98-633586d322f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:44.467816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa6783-86f5-457e-af98-633586d322f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:44.468084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aa6783-86f5-457e-af98-633586d322f2 admin admin] PATH_INFO: `/v3/projects/d6018b92484a4751b5493e4240a0992a/users/b390472c536b4ddda83181e085eab456/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:44.486479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42aa6783-86f5-457e-af98-633586d322f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6018b92484a4751b5493e4240a0992a, user_id=b390472c536b4ddda83181e085eab456, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:44.488893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42aa6783-86f5-457e-af98-633586d322f2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:44.618374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 966/1949] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:44 2022] PUT /identity/v3/projects/d6018b92484a4751b5493e4240a0992a/users/b390472c536b4ddda83181e085eab456/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:44.630010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28502d8-3479-45df-b65f-098d59dd340f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:44.630299 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28502d8-3479-45df-b65f-098d59dd340f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:44.630577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28502d8-3479-45df-b65f-098d59dd340f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:44.664484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c28502d8-3479-45df-b65f-098d59dd340f None None] MFA Rules not processed for user `b390472c536b4ddda83181e085eab456`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:44.666480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c28502d8-3479-45df-b65f-098d59dd340f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:44.759922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 984/1950] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:44 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:44.765160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aa6783-86f5-457e-af98-633586d322f2 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:44.829521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:44.843893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aa6783-86f5-457e-af98-633586d322f2 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:44.848602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87c4fae-e51f-4741-bd06-7eb90489e6fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:44.849261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87c4fae-e51f-4741-bd06-7eb90489e6fd admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:44.849695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87c4fae-e51f-4741-bd06-7eb90489e6fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:44.849973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87c4fae-e51f-4741-bd06-7eb90489e6fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:44.850600 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87c4fae-e51f-4741-bd06-7eb90489e6fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:44.852560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87c4fae-e51f-4741-bd06-7eb90489e6fd admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:44.942652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28502d8-3479-45df-b65f-098d59dd340f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:44.944009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c28502d8-3479-45df-b65f-098d59dd340f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:44.958781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28502d8-3479-45df-b65f-098d59dd340f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:44.966538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:44.967207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:44.967623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:44.967822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] PATH_INFO: `/v3/projects/f72c99bf3d5e4f688925776aa2952fa1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:44.969212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f72c99bf3d5e4f688925776aa2952fa1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:44.971895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:44.980807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 967/1951] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:44 2022] POST /identity/v3/projects => generated 389 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:44.981450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87c4fae-e51f-4741-bd06-7eb90489e6fd admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.053681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87c4fae-e51f-4741-bd06-7eb90489e6fd admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.059298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25bba40-3051-477f-b9b5-e8936d76e91b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.059919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25bba40-3051-477f-b9b5-e8936d76e91b admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.060203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25bba40-3051-477f-b9b5-e8936d76e91b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.060484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25bba40-3051-477f-b9b5-e8936d76e91b admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.061116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25bba40-3051-477f-b9b5-e8936d76e91b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.063101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25bba40-3051-477f-b9b5-e8936d76e91b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.091038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f72c99bf3d5e4f688925776aa2952fa1', 'request_id': 'req-734ebd3a-a7be-4b95-bc35-33a644e76fb6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:45.096780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f72c99bf3d5e4f688925776aa2952fa1', 'request_id': 'req-734ebd3a-a7be-4b95-bc35-33a644e76fb6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:45.163023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 968/1952] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:44 2022] POST /identity/v3/users => generated 301 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:45.166810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25bba40-3051-477f-b9b5-e8936d76e91b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.177655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25bba40-3051-477f-b9b5-e8936d76e91b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.180720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f72c99bf3d5e4f688925776aa2952fa1', 'request_id': 'req-734ebd3a-a7be-4b95-bc35-33a644e76fb6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:45.194102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] The token cache is being invalidate because project f72c99bf3d5e4f688925776aa2952fa1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:45.194632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:45.195535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-734ebd3a-a7be-4b95-bc35-33a644e76fb6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:45.199684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 985/1953] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:44 2022] DELETE /identity/v3/projects/f72c99bf3d5e4f688925776aa2952fa1 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:45.203807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d85db6d-8d04-4471-a327-9a4a9a425ea5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.204272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d85db6d-8d04-4471-a327-9a4a9a425ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.204694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d85db6d-8d04-4471-a327-9a4a9a425ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.234704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2d85db6d-8d04-4471-a327-9a4a9a425ea5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:45.236042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2d85db6d-8d04-4471-a327-9a4a9a425ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:45.244102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.244718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.244996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.245268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.245973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.247571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.249667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:45.257549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 969/1954] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:45 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:45.262293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.263940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:45.275739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f4481-2957-41f6-bb2b-6c05eda3e84e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.280653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dff930-abee-4dee-9ffb-06f7aa7d9c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.281482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dff930-abee-4dee-9ffb-06f7aa7d9c6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.282123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dff930-abee-4dee-9ffb-06f7aa7d9c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.282572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dff930-abee-4dee-9ffb-06f7aa7d9c6e admin admin] PATH_INFO: `/v3/projects/641cf939b08c49a79eab937809054002/users/3332a3f169c14048bef41209d95e8133/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.299398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dff930-abee-4dee-9ffb-06f7aa7d9c6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=641cf939b08c49a79eab937809054002, user_id=3332a3f169c14048bef41209d95e8133, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.302298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dff930-abee-4dee-9ffb-06f7aa7d9c6e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.302547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 986/1955] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:45.311010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1c6b0f-09aa-4d55-9fa2-795b1c7bfee6 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.311241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1c6b0f-09aa-4d55-9fa2-795b1c7bfee6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.311628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1c6b0f-09aa-4d55-9fa2-795b1c7bfee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.330579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7a1c6b0f-09aa-4d55-9fa2-795b1c7bfee6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:45.331773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7a1c6b0f-09aa-4d55-9fa2-795b1c7bfee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:45.339896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 987/1956] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:45.343867 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1c6b0f-09aa-4d55-9fa2-795b1c7bfee6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.355736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1c6b0f-09aa-4d55-9fa2-795b1c7bfee6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.361300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.361945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.362221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.362487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.363138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:45.363434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.365046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.381699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 988/1957] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:14:45.382180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 970/1958] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:45 2022] PUT /identity/v3/projects/641cf939b08c49a79eab937809054002/users/3332a3f169c14048bef41209d95e8133/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:45.387116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.390199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dff930-abee-4dee-9ffb-06f7aa7d9c6e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.396303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0aade3-1e2f-4b32-b323-739de01cd5e0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.444223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:45.460165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e0dd69-2358-40b3-b2bd-2cb92c078b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.460786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e0dd69-2358-40b3-b2bd-2cb92c078b51 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.461075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e0dd69-2358-40b3-b2bd-2cb92c078b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.461354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e0dd69-2358-40b3-b2bd-2cb92c078b51 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.462049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e0dd69-2358-40b3-b2bd-2cb92c078b51 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.464399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e0dd69-2358-40b3-b2bd-2cb92c078b51 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.468077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dff930-abee-4dee-9ffb-06f7aa7d9c6e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.478453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0a51c2-80bf-43d9-9a47-90aaf79d4822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.479244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a51c2-80bf-43d9-9a47-90aaf79d4822 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.479708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a51c2-80bf-43d9-9a47-90aaf79d4822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.480182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0a51c2-80bf-43d9-9a47-90aaf79d4822 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.481692 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0a51c2-80bf-43d9-9a47-90aaf79d4822 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.488344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0a51c2-80bf-43d9-9a47-90aaf79d4822 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.497666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 971/1959] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:45 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:14:45.498840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0a51c2-80bf-43d9-9a47-90aaf79d4822 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.508745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0a51c2-80bf-43d9-9a47-90aaf79d4822 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.514164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdde364-2f10-42a0-97b1-fc49d15c5753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.514816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdde364-2f10-42a0-97b1-fc49d15c5753 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.515092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdde364-2f10-42a0-97b1-fc49d15c5753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.515360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdde364-2f10-42a0-97b1-fc49d15c5753 admin admin] PATH_INFO: `/v3/projects/641cf939b08c49a79eab937809054002/users/3332a3f169c14048bef41209d95e8133/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.517381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdde364-2f10-42a0-97b1-fc49d15c5753 admin admin] RBAC: Authorizing `identity:create_grant(project_id=641cf939b08c49a79eab937809054002, user_id=3332a3f169c14048bef41209d95e8133, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.520061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdde364-2f10-42a0-97b1-fc49d15c5753 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.531203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 972/1960] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:45 2022] PUT /identity/v3/projects/641cf939b08c49a79eab937809054002/users/3332a3f169c14048bef41209d95e8133/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:45.542310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f7402-8564-4e52-a5de-c65f5b40b5a3 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.542595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f7402-8564-4e52-a5de-c65f5b40b5a3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.542897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6f7402-8564-4e52-a5de-c65f5b40b5a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.570662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-cc6f7402-8564-4e52-a5de-c65f5b40b5a3 None None] MFA Rules not processed for user `3332a3f169c14048bef41209d95e8133`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:45.571878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cc6f7402-8564-4e52-a5de-c65f5b40b5a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:45.622883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 989/1961] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:45 2022] POST /identity/v3/projects => generated 373 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:45.627081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e0dd69-2358-40b3-b2bd-2cb92c078b51 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.679460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 973/1962] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:45 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:45.690544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f7402-8564-4e52-a5de-c65f5b40b5a3 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.692664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cc6f7402-8564-4e52-a5de-c65f5b40b5a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:45.722776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e0dd69-2358-40b3-b2bd-2cb92c078b51 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.728005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f732b9-a6b1-4a98-b5a1-ea6159f42365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.728590 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f732b9-a6b1-4a98-b5a1-ea6159f42365 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.728948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f732b9-a6b1-4a98-b5a1-ea6159f42365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.729213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f732b9-a6b1-4a98-b5a1-ea6159f42365 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.729857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f732b9-a6b1-4a98-b5a1-ea6159f42365 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.731892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f732b9-a6b1-4a98-b5a1-ea6159f42365 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.765338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6f7402-8564-4e52-a5de-c65f5b40b5a3 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.812741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502e2591-e126-4ff1-9b9c-9e470a07eee3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.813417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e2591-e126-4ff1-9b9c-9e470a07eee3 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.813738 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e2591-e126-4ff1-9b9c-9e470a07eee3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.814019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e2591-e126-4ff1-9b9c-9e470a07eee3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.872044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502e2591-e126-4ff1-9b9c-9e470a07eee3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.873493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502e2591-e126-4ff1-9b9c-9e470a07eee3 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.882854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 974/1963] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:45 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:45.909515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 990/1964] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:45 2022] POST /identity/v3/users => generated 287 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:14:45.913538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502e2591-e126-4ff1-9b9c-9e470a07eee3 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.924243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502e2591-e126-4ff1-9b9c-9e470a07eee3 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.929685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d5b04-8f1c-4a51-b587-5554a6e8b270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.930296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5b04-8f1c-4a51-b587-5554a6e8b270 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.930574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5b04-8f1c-4a51-b587-5554a6e8b270 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.933240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d5b04-8f1c-4a51-b587-5554a6e8b270 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.933240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d5b04-8f1c-4a51-b587-5554a6e8b270 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.933240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d5b04-8f1c-4a51-b587-5554a6e8b270 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.944065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 975/1965] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:45.948101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f732b9-a6b1-4a98-b5a1-ea6159f42365 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.959035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f732b9-a6b1-4a98-b5a1-ea6159f42365 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.963987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602ffa25-60b8-498c-990e-af4c1fc90d41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:45.964858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602ffa25-60b8-498c-990e-af4c1fc90d41 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:45.965142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602ffa25-60b8-498c-990e-af4c1fc90d41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:45.965435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602ffa25-60b8-498c-990e-af4c1fc90d41 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:45.966128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602ffa25-60b8-498c-990e-af4c1fc90d41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:45.967737 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602ffa25-60b8-498c-990e-af4c1fc90d41 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:45.978206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 991/1966] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:45.983512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d5b04-8f1c-4a51-b587-5554a6e8b270 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:45.994057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d5b04-8f1c-4a51-b587-5554a6e8b270 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:45.999757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7594fd7-05dd-4d0d-b38b-9e19297c7fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:46.000430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7594fd7-05dd-4d0d-b38b-9e19297c7fd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:46.000716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7594fd7-05dd-4d0d-b38b-9e19297c7fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:46.001023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7594fd7-05dd-4d0d-b38b-9e19297c7fd6 admin admin] PATH_INFO: `/v3/projects/a252c620742a4f6b9fdaafb4cb81f10f/users/5be22eb85b3b4828a36edc6e1fcc05f7/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:46.016792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7594fd7-05dd-4d0d-b38b-9e19297c7fd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a252c620742a4f6b9fdaafb4cb81f10f, user_id=5be22eb85b3b4828a36edc6e1fcc05f7, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:46.019552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7594fd7-05dd-4d0d-b38b-9e19297c7fd6 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:46.035206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602ffa25-60b8-498c-990e-af4c1fc90d41 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:46.036777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:46.040530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:46.046519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602ffa25-60b8-498c-990e-af4c1fc90d41 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:46.051008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:46.053741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:46.053741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:46.053741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] PATH_INFO: `/v3/users/b390472c536b4ddda83181e085eab456` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:46.053741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b390472c536b4ddda83181e085eab456)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:46.055278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:46.111054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 976/1967] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:45 2022] PUT /identity/v3/projects/a252c620742a4f6b9fdaafb4cb81f10f/users/5be22eb85b3b4828a36edc6e1fcc05f7/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:46.116063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3cf842-ec4d-4a07-8506-a9940b178082 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:46.116374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3cf842-ec4d-4a07-8506-a9940b178082 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:46.116644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3cf842-ec4d-4a07-8506-a9940b178082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:46.136385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ae3cf842-ec4d-4a07-8506-a9940b178082 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:46.137564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ae3cf842-ec4d-4a07-8506-a9940b178082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:46.198883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 977/1968] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:46.208422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3cf842-ec4d-4a07-8506-a9940b178082 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:46.209837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ae3cf842-ec4d-4a07-8506-a9940b178082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:46.264868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3cf842-ec4d-4a07-8506-a9940b178082 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:46.268422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c443ad-6c74-4a8e-80ff-1db27282dea9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:46.269024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c443ad-6c74-4a8e-80ff-1db27282dea9 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:46.269301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c443ad-6c74-4a8e-80ff-1db27282dea9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:46.269571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c443ad-6c74-4a8e-80ff-1db27282dea9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:46.278176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c443ad-6c74-4a8e-80ff-1db27282dea9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:46.279594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c443ad-6c74-4a8e-80ff-1db27282dea9 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:46.288951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 978/1969] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:14:46.296830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b390472c536b4ddda83181e085eab456'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:46.297516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b390472c536b4ddda83181e085eab456'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:46.311303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b390472c536b4ddda83181e085eab456'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:46.384327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b390472c536b4ddda83181e085eab456'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:46.391862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b390472c536b4ddda83181e085eab456'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:46.397799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 992/1970] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:46 2022] DELETE /identity/v3/users/b390472c536b4ddda83181e085eab456 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:14:46.402843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d64261-e18a-44da-ada0-71727985a303 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:46.403196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d64261-e18a-44da-ada0-71727985a303 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:46.403535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d64261-e18a-44da-ada0-71727985a303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:46.422944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f4d64261-e18a-44da-ada0-71727985a303 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:46.424144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f4d64261-e18a-44da-ada0-71727985a303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:46.441100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:46.485127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 979/1971] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:46.494832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:46.496907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:46.553967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23bc4af-167b-4f54-8695-4b1996dd14d1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:46.557807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec699e-6366-4c53-b6da-1af2d47b0777 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:46.558207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec699e-6366-4c53-b6da-1af2d47b0777 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:46.559066 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec699e-6366-4c53-b6da-1af2d47b0777 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:46.559341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec699e-6366-4c53-b6da-1af2d47b0777 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:46.569512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ec699e-6366-4c53-b6da-1af2d47b0777 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:46.571027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ec699e-6366-4c53-b6da-1af2d47b0777 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:46.582647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 993/1972] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:47.292244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d64261-e18a-44da-ada0-71727985a303 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:47.301749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f4d64261-e18a-44da-ada0-71727985a303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:47.306207 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d64261-e18a-44da-ada0-71727985a303 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:47.311206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:47.311907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:47.312294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:47.312668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] PATH_INFO: `/v3/projects/d6018b92484a4751b5493e4240a0992a` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:47.314082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6018b92484a4751b5493e4240a0992a)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:47.316088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:47.431589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6018b92484a4751b5493e4240a0992a', 'request_id': 'req-97b20f1e-e530-416f-a404-0fb791b5f323'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:47.522195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6018b92484a4751b5493e4240a0992a', 'request_id': 'req-97b20f1e-e530-416f-a404-0fb791b5f323'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:47.529525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6018b92484a4751b5493e4240a0992a', 'request_id': 'req-97b20f1e-e530-416f-a404-0fb791b5f323'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:47.540921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] The token cache is being invalidate because project d6018b92484a4751b5493e4240a0992a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:47.541215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:47.541906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-97b20f1e-e530-416f-a404-0fb791b5f323 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:47.543695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 980/1973] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:47 2022] DELETE /identity/v3/projects/d6018b92484a4751b5493e4240a0992a => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:47.547687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec699e-6366-4c53-b6da-1af2d47b0777 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:47.557561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec699e-6366-4c53-b6da-1af2d47b0777 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:47.615545 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:47.616171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:47.616444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:47.616711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] PATH_INFO: `/v3/users/3332a3f169c14048bef41209d95e8133` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:47.617862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3332a3f169c14048bef41209d95e8133)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:47.619756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:47.640239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:47.867969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3332a3f169c14048bef41209d95e8133'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:47.868739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3332a3f169c14048bef41209d95e8133'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:47.884252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3332a3f169c14048bef41209d95e8133'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:47.924246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9f28d2-34da-4608-bd1f-57adfd7f62ff None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:47.924753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9f28d2-34da-4608-bd1f-57adfd7f62ff None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:47.926058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9f28d2-34da-4608-bd1f-57adfd7f62ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:47.969548 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-7d9f28d2-34da-4608-bd1f-57adfd7f62ff None None] MFA Rules not processed for user `5be22eb85b3b4828a36edc6e1fcc05f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:47.970863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7d9f28d2-34da-4608-bd1f-57adfd7f62ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:48.014209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3332a3f169c14048bef41209d95e8133'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:48.029030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3332a3f169c14048bef41209d95e8133'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:48.037297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 994/1974] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:47 2022] DELETE /identity/v3/users/3332a3f169c14048bef41209d95e8133 => generated 0 bytes in 490 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:48.071287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 981/1975] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:48.087971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:48.089724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:48.157094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d6c937-7518-4342-bc79-21ba72886c36 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:48.160725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47845e0-bceb-45da-adc2-975da80ea16c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:48.161335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47845e0-bceb-45da-adc2-975da80ea16c service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:48.161636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47845e0-bceb-45da-adc2-975da80ea16c service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:48.161916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47845e0-bceb-45da-adc2-975da80ea16c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:48.219051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47845e0-bceb-45da-adc2-975da80ea16c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:48.220564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47845e0-bceb-45da-adc2-975da80ea16c service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:48.230102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 995/1976] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:48 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:48.502070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9f28d2-34da-4608-bd1f-57adfd7f62ff None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:48.503550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7d9f28d2-34da-4608-bd1f-57adfd7f62ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:48.517406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9f28d2-34da-4608-bd1f-57adfd7f62ff None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:48.551450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:48.575402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:48.576031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:48.576309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:48.576579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] PATH_INFO: `/v3/projects/641cf939b08c49a79eab937809054002` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:48.577725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=641cf939b08c49a79eab937809054002)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:48.579616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:48.699880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '641cf939b08c49a79eab937809054002', 'request_id': 'req-1344e1df-da84-4967-a905-16483769f6f5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:48.810468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '641cf939b08c49a79eab937809054002', 'request_id': 'req-1344e1df-da84-4967-a905-16483769f6f5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:48.816581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '641cf939b08c49a79eab937809054002', 'request_id': 'req-1344e1df-da84-4967-a905-16483769f6f5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:48.829509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] The token cache is being invalidate because project 641cf939b08c49a79eab937809054002 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:14:48.829829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:48.830967 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1344e1df-da84-4967-a905-16483769f6f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:48.832683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 982/1977] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:14:48 2022] DELETE /identity/v3/projects/641cf939b08c49a79eab937809054002 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:48.837536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03d4bb-4f8c-4df8-9a66-c8a2e547f9d0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:48.838062 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03d4bb-4f8c-4df8-9a66-c8a2e547f9d0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:48.838162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03d4bb-4f8c-4df8-9a66-c8a2e547f9d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:48.860018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8d03d4bb-4f8c-4df8-9a66-c8a2e547f9d0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:48.861228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8d03d4bb-4f8c-4df8-9a66-c8a2e547f9d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:48.915112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:48.934995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 996/1978] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:48.943283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18154b8e-4d24-4557-87fd-f4f8b17817ad None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:48.943587 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18154b8e-4d24-4557-87fd-f4f8b17817ad None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:48.944130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18154b8e-4d24-4557-87fd-f4f8b17817ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:48.966220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-18154b8e-4d24-4557-87fd-f4f8b17817ad None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:48.967422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-18154b8e-4d24-4557-87fd-f4f8b17817ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:48.975447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 983/1979] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:48.979715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d03d4bb-4f8c-4df8-9a66-c8a2e547f9d0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:48.989749 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d03d4bb-4f8c-4df8-9a66-c8a2e547f9d0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:48.994342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:48.994947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:48.995228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:48.995503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:48.996301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:14:48.996610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:48.998231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:49.010697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 997/1980] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:14:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:14:49.015989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18154b8e-4d24-4557-87fd-f4f8b17817ad None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:49.025958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18154b8e-4d24-4557-87fd-f4f8b17817ad None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:49.031216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cccee82-57d7-41b9-a482-ec9849e99591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:49.031825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cccee82-57d7-41b9-a482-ec9849e99591 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:49.032135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cccee82-57d7-41b9-a482-ec9849e99591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:49.032406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cccee82-57d7-41b9-a482-ec9849e99591 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:49.033031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cccee82-57d7-41b9-a482-ec9849e99591 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:49.034987 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cccee82-57d7-41b9-a482-ec9849e99591 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:49.201456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:49.202024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 984/1981] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:49 2022] POST /identity/v3/projects => generated 367 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:49.215371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b19ffe-ab74-4bf0-8a6b-69d12101ba59 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:49.283286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ac76c-c0ac-4949-8e9d-7ec69f616097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:49.283911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ac76c-c0ac-4949-8e9d-7ec69f616097 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:49.284191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ac76c-c0ac-4949-8e9d-7ec69f616097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:49.284469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07ac76c-c0ac-4949-8e9d-7ec69f616097 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:49.285128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ac76c-c0ac-4949-8e9d-7ec69f616097 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:49.287112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07ac76c-c0ac-4949-8e9d-7ec69f616097 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:49.437140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 998/1982] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:49 2022] POST /identity/v3/users => generated 284 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:49.441244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cccee82-57d7-41b9-a482-ec9849e99591 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:49.452092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cccee82-57d7-41b9-a482-ec9849e99591 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:49.456694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c652a1b-4e03-4602-96b8-237de7b0d683 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:49.457327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c652a1b-4e03-4602-96b8-237de7b0d683 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:49.457644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c652a1b-4e03-4602-96b8-237de7b0d683 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:49.457950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c652a1b-4e03-4602-96b8-237de7b0d683 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:49.458593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c652a1b-4e03-4602-96b8-237de7b0d683 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:49.460218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c652a1b-4e03-4602-96b8-237de7b0d683 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:49.470647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 985/1983] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:49.474734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ac76c-c0ac-4949-8e9d-7ec69f616097 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:49.484393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07ac76c-c0ac-4949-8e9d-7ec69f616097 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:49.488862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070020ec-bc8f-4c4b-ae1c-4891e6b17af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:49.489465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070020ec-bc8f-4c4b-ae1c-4891e6b17af7 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:49.489811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070020ec-bc8f-4c4b-ae1c-4891e6b17af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:49.490114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070020ec-bc8f-4c4b-ae1c-4891e6b17af7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:49.490767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070020ec-bc8f-4c4b-ae1c-4891e6b17af7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:49.492409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070020ec-bc8f-4c4b-ae1c-4891e6b17af7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:49.503297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 999/1984] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:49.507480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c652a1b-4e03-4602-96b8-237de7b0d683 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:49.518086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c652a1b-4e03-4602-96b8-237de7b0d683 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:49.522728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86b72cd-73da-40ef-a9bb-a18e97469c5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:49.523374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86b72cd-73da-40ef-a9bb-a18e97469c5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:49.523656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86b72cd-73da-40ef-a9bb-a18e97469c5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:49.523931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86b72cd-73da-40ef-a9bb-a18e97469c5f admin admin] PATH_INFO: `/v3/projects/4e1adff6c1f74da19f8dab38ad35e5af/users/46c59312834e4f8aa41b98f1130f658d/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:49.538238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86b72cd-73da-40ef-a9bb-a18e97469c5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e1adff6c1f74da19f8dab38ad35e5af, user_id=46c59312834e4f8aa41b98f1130f658d, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:49.540875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86b72cd-73da-40ef-a9bb-a18e97469c5f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:49.613580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 986/1985] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:49 2022] PUT /identity/v3/projects/4e1adff6c1f74da19f8dab38ad35e5af/users/46c59312834e4f8aa41b98f1130f658d/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:49.624337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7756ddf8-7bc7-4172-b7ed-0e3bd6faaad5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:49.624706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7756ddf8-7bc7-4172-b7ed-0e3bd6faaad5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:49.625036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7756ddf8-7bc7-4172-b7ed-0e3bd6faaad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:49.652635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7756ddf8-7bc7-4172-b7ed-0e3bd6faaad5 None None] MFA Rules not processed for user `46c59312834e4f8aa41b98f1130f658d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:49.653849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7756ddf8-7bc7-4172-b7ed-0e3bd6faaad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:49.717756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1000/1986] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:49 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:49.722426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86b72cd-73da-40ef-a9bb-a18e97469c5f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:49.783209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86b72cd-73da-40ef-a9bb-a18e97469c5f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:49.787807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799c3735-db55-45c6-a23e-e113cd0b847d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:49.788410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799c3735-db55-45c6-a23e-e113cd0b847d admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:49.788693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799c3735-db55-45c6-a23e-e113cd0b847d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:49.788967 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799c3735-db55-45c6-a23e-e113cd0b847d admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:49.789581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799c3735-db55-45c6-a23e-e113cd0b847d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:49.791508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799c3735-db55-45c6-a23e-e113cd0b847d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:49.797334 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:49.902567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 987/1987] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:14:49 2022] POST /identity/v3/projects => generated 367 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:49.903663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7756ddf8-7bc7-4172-b7ed-0e3bd6faaad5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:49.920412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7756ddf8-7bc7-4172-b7ed-0e3bd6faaad5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:49.963871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:49.977363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36675ec2-af3a-4d06-94b0-1ed3dbc6ed59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:49.978010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36675ec2-af3a-4d06-94b0-1ed3dbc6ed59 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:49.984701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36675ec2-af3a-4d06-94b0-1ed3dbc6ed59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:49.984701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36675ec2-af3a-4d06-94b0-1ed3dbc6ed59 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:49.984701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36675ec2-af3a-4d06-94b0-1ed3dbc6ed59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:49.984701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36675ec2-af3a-4d06-94b0-1ed3dbc6ed59 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:50.076452 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1001/1988] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:14:49 2022] POST /identity/v3/users => generated 290 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:14:50.080407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799c3735-db55-45c6-a23e-e113cd0b847d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:50.091359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799c3735-db55-45c6-a23e-e113cd0b847d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:50.095936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415c1139-7258-4da1-b488-fda8afda7e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:50.096534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415c1139-7258-4da1-b488-fda8afda7e65 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:50.096814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415c1139-7258-4da1-b488-fda8afda7e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:50.097086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415c1139-7258-4da1-b488-fda8afda7e65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:50.097761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415c1139-7258-4da1-b488-fda8afda7e65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:50.099357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415c1139-7258-4da1-b488-fda8afda7e65 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:50.109893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 988/1989] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:50.117890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36675ec2-af3a-4d06-94b0-1ed3dbc6ed59 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:50.127496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36675ec2-af3a-4d06-94b0-1ed3dbc6ed59 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:50.132000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ccecdd-4b04-4090-96a7-03bc32a1bb25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:50.132626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ccecdd-4b04-4090-96a7-03bc32a1bb25 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:50.132905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ccecdd-4b04-4090-96a7-03bc32a1bb25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:50.133177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ccecdd-4b04-4090-96a7-03bc32a1bb25 admin admin] PATH_INFO: `/v3/projects/cde899656eeb43828af519427bbcee68/users/73e8f158e9f446dba709cee4d528cb41/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:50.147785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ccecdd-4b04-4090-96a7-03bc32a1bb25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cde899656eeb43828af519427bbcee68, user_id=73e8f158e9f446dba709cee4d528cb41, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:50.150465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ccecdd-4b04-4090-96a7-03bc32a1bb25 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:50.263460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1002/1990] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:50 2022] PUT /identity/v3/projects/cde899656eeb43828af519427bbcee68/users/73e8f158e9f446dba709cee4d528cb41/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:50.267538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415c1139-7258-4da1-b488-fda8afda7e65 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:50.278993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415c1139-7258-4da1-b488-fda8afda7e65 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:50.339458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d9d37c-f67d-49d7-9a9e-af886d173bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:50.340582 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d9d37c-f67d-49d7-9a9e-af886d173bf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:50.340864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d9d37c-f67d-49d7-9a9e-af886d173bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:50.341133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d9d37c-f67d-49d7-9a9e-af886d173bf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:50.341799 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d9d37c-f67d-49d7-9a9e-af886d173bf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:50.343395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d9d37c-f67d-49d7-9a9e-af886d173bf4 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:50.357568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 989/1991] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:14:50 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:14:50.361637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ccecdd-4b04-4090-96a7-03bc32a1bb25 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:50.372194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ccecdd-4b04-4090-96a7-03bc32a1bb25 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:50.377757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:50.378466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:50.378744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:50.379361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] PATH_INFO: `/v3/projects/cde899656eeb43828af519427bbcee68/users/73e8f158e9f446dba709cee4d528cb41/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:50.380901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cde899656eeb43828af519427bbcee68, user_id=73e8f158e9f446dba709cee4d528cb41, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:50.383569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:50.394570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1003/1992] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:14:50 2022] PUT /identity/v3/projects/cde899656eeb43828af519427bbcee68/users/73e8f158e9f446dba709cee4d528cb41/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:50.406266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317892ce-dbb9-4afd-9477-cd70fe77b2dd None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:50.406742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317892ce-dbb9-4afd-9477-cd70fe77b2dd None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:50.414720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317892ce-dbb9-4afd-9477-cd70fe77b2dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:50.443168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-317892ce-dbb9-4afd-9477-cd70fe77b2dd None None] MFA Rules not processed for user `73e8f158e9f446dba709cee4d528cb41`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:50.444393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-317892ce-dbb9-4afd-9477-cd70fe77b2dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:50.516427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 990/1993] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:50 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:50.528392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:50.531282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:50.587773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b6159f-6559-4d5b-8820-74ccc3c0b200 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:50.592281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3074c163-c36a-470e-b49f-07dbedb28d0c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:50.592885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074c163-c36a-470e-b49f-07dbedb28d0c service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:50.593155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074c163-c36a-470e-b49f-07dbedb28d0c service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:50.593421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3074c163-c36a-470e-b49f-07dbedb28d0c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:50.602130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3074c163-c36a-470e-b49f-07dbedb28d0c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:50.603579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3074c163-c36a-470e-b49f-07dbedb28d0c service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:50.612705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1004/1994] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:50 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:50.906203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317892ce-dbb9-4afd-9477-cd70fe77b2dd None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:50.915794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317892ce-dbb9-4afd-9477-cd70fe77b2dd None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:50.919384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:50.919986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:50.920264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:50.920541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:50.976994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:50.979321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:50.980778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:50.989750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 991/1995] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:50 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:14:54.259263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c69b22-c8d4-44ae-b6d5-092948997276 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:54.259745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c69b22-c8d4-44ae-b6d5-092948997276 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:54.259838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c69b22-c8d4-44ae-b6d5-092948997276 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:54.278592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b8c69b22-c8d4-44ae-b6d5-092948997276 None None] MFA Rules not processed for user `5be22eb85b3b4828a36edc6e1fcc05f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:54.279776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b8c69b22-c8d4-44ae-b6d5-092948997276 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:54.334125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1005/1996] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:14:58.777007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:58.779015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:58.857498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5836b0b4-4cd2-428c-8e12-d1236b95fa7b service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:58.862142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:58.862776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:58.863055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:58.863341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] PATH_INFO: `/v3/users/58170b7e613b4f43b7dd7ed8ad27a3fd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:58.864489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] RBAC: Authorizing `identity:delete_user(user_id=58170b7e613b4f43b7dd7ed8ad27a3fd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:58.866444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:59.064781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58170b7e613b4f43b7dd7ed8ad27a3fd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:59.087937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58170b7e613b4f43b7dd7ed8ad27a3fd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:59.088879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58170b7e613b4f43b7dd7ed8ad27a3fd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:59.156950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58170b7e613b4f43b7dd7ed8ad27a3fd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:59.165861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58170b7e613b4f43b7dd7ed8ad27a3fd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:14:59.173353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 992/1997] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:14:58 2022] DELETE /identity/v3/users/58170b7e613b4f43b7dd7ed8ad27a3fd => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:14:59.178400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9fd595-3ff9-44b7-ba63-df2833ba84be None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:59.178880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9fd595-3ff9-44b7-ba63-df2833ba84be None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:59.179140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9fd595-3ff9-44b7-ba63-df2833ba84be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:59.202575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9c9fd595-3ff9-44b7-ba63-df2833ba84be None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:14:59.203744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9c9fd595-3ff9-44b7-ba63-df2833ba84be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:59.273164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:14:59.299619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1006/1998] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:14:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:14:59.309871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:14:59.311544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:14:59.370756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d3f323-f63a-4623-a62b-3182027a4b12 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:14:59.374711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:14:59.375071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:14:59.375348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:14:59.375614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:14:59.384885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:14:59.386370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:14:59.397023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 993/1999] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:14:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:00.125702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9fd595-3ff9-44b7-ba63-df2833ba84be None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:00.127579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9c9fd595-3ff9-44b7-ba63-df2833ba84be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:00.145128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9fd595-3ff9-44b7-ba63-df2833ba84be None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:00.155680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:00.157285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:00.157863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:00.158575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] PATH_INFO: `/v3/projects/dc378b4b85c244699e7ed199fd73d1f4` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:00.160204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc378b4b85c244699e7ed199fd73d1f4)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:00.162730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:00.323619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:00.326846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:00.348183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc378b4b85c244699e7ed199fd73d1f4', 'request_id': 'req-6b743a78-0d3f-4dc9-92d6-7f441617361d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.354563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22842a4a-cc2c-434b-8389-fcb1672820b2 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:00.356862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc378b4b85c244699e7ed199fd73d1f4', 'request_id': 'req-6b743a78-0d3f-4dc9-92d6-7f441617361d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.372371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:00.448284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc378b4b85c244699e7ed199fd73d1f4', 'request_id': 'req-6b743a78-0d3f-4dc9-92d6-7f441617361d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.457342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:00.458140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:00.458489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:00.458801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] PATH_INFO: `/v3/users/46c59312834e4f8aa41b98f1130f658d` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:00.460059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46c59312834e4f8aa41b98f1130f658d)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:00.467316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:00.472192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] The token cache is being invalidate because project dc378b4b85c244699e7ed199fd73d1f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:15:00.472487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.473166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6b743a78-0d3f-4dc9-92d6-7f441617361d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.475532 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1007/2000] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:15:00 2022] DELETE /identity/v3/projects/dc378b4b85c244699e7ed199fd73d1f4 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:00.502726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b811afb-6fbb-42a6-a2ef-2f9e4dd4870d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:00.505938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b811afb-6fbb-42a6-a2ef-2f9e4dd4870d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:00.506411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b811afb-6fbb-42a6-a2ef-2f9e4dd4870d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:00.538454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2b811afb-6fbb-42a6-a2ef-2f9e4dd4870d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:00.539774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2b811afb-6fbb-42a6-a2ef-2f9e4dd4870d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:00.550387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1008/2001] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:00.559991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a170761-f9a6-4bd4-b0f2-d9d2b6798382 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:00.559991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a170761-f9a6-4bd4-b0f2-d9d2b6798382 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:00.560227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a170761-f9a6-4bd4-b0f2-d9d2b6798382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:00.586592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4a170761-f9a6-4bd4-b0f2-d9d2b6798382 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:00.587905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4a170761-f9a6-4bd4-b0f2-d9d2b6798382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:00.601412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1009/2002] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:00.603296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a170761-f9a6-4bd4-b0f2-d9d2b6798382 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:00.614526 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a170761-f9a6-4bd4-b0f2-d9d2b6798382 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:00.619305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:00.620114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:00.620576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:00.621015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:00.621721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:15:00.622101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:00.623801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:00.649343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1010/2003] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:15:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:15:00.654098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:00.663865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc19aad-4833-4024-9616-28865a53e042 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:00.669158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:00.669822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:00.670107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:00.670383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:00.671011 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:00.672977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:00.732075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46c59312834e4f8aa41b98f1130f658d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.746532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46c59312834e4f8aa41b98f1130f658d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.747242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46c59312834e4f8aa41b98f1130f658d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.798257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1011/2004] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:15:00 2022] POST /identity/v3/projects => generated 365 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:15:00.802534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:00.804181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:00.820044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46c59312834e4f8aa41b98f1130f658d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.830005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-463fd642-5a7c-4cf4-b8d7-e19763686f49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46c59312834e4f8aa41b98f1130f658d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:00.844810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 994/2005] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:15:00 2022] DELETE /identity/v3/users/46c59312834e4f8aa41b98f1130f658d => generated 0 bytes in 522 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:00.850448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54156a41-d4c1-4866-9979-be8cba1c13f5 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:00.850448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54156a41-d4c1-4866-9979-be8cba1c13f5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:00.850448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54156a41-d4c1-4866-9979-be8cba1c13f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:00.883014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-54156a41-d4c1-4866-9979-be8cba1c13f5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:00.884166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-54156a41-d4c1-4866-9979-be8cba1c13f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:00.906199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72f752e-f7df-4f1f-b5d2-e33b05ea5abd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:00.940994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:00.951487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 995/2006] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:00.961467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54156a41-d4c1-4866-9979-be8cba1c13f5 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:00.963093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-54156a41-d4c1-4866-9979-be8cba1c13f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:00.978221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe00215-cd3b-4a72-98ef-c40f0a8283ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:00.982221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe00215-cd3b-4a72-98ef-c40f0a8283ac admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:00.982221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe00215-cd3b-4a72-98ef-c40f0a8283ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:00.982221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe00215-cd3b-4a72-98ef-c40f0a8283ac admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:00.982221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe00215-cd3b-4a72-98ef-c40f0a8283ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:00.983065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe00215-cd3b-4a72-98ef-c40f0a8283ac admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:01.022365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54156a41-d4c1-4866-9979-be8cba1c13f5 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:01.026189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be968d9-9575-4d39-9399-8fb0e197f0ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:01.027497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be968d9-9575-4d39-9399-8fb0e197f0ac service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:01.027855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be968d9-9575-4d39-9399-8fb0e197f0ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:01.028207 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be968d9-9575-4d39-9399-8fb0e197f0ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:01.037504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be968d9-9575-4d39-9399-8fb0e197f0ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:01.039044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be968d9-9575-4d39-9399-8fb0e197f0ac service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:01.048471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 996/2007] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:15:01.121994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1012/2008] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:15:00 2022] POST /identity/v3/users => generated 283 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:15:01.126684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be968d9-9575-4d39-9399-8fb0e197f0ac service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:01.138336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be968d9-9575-4d39-9399-8fb0e197f0ac service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:01.143816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48857b3e-39eb-4664-bcec-effc208b70e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:01.144430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48857b3e-39eb-4664-bcec-effc208b70e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:01.144711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48857b3e-39eb-4664-bcec-effc208b70e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:01.144982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48857b3e-39eb-4664-bcec-effc208b70e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:01.145676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48857b3e-39eb-4664-bcec-effc208b70e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:01.147304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48857b3e-39eb-4664-bcec-effc208b70e3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:01.158161 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 997/2009] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:01.166932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe00215-cd3b-4a72-98ef-c40f0a8283ac admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:01.177071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe00215-cd3b-4a72-98ef-c40f0a8283ac admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:01.181821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37db3e1-f53a-43a2-9152-2cf952c943c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:01.182420 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37db3e1-f53a-43a2-9152-2cf952c943c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:01.182708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37db3e1-f53a-43a2-9152-2cf952c943c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:01.182984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37db3e1-f53a-43a2-9152-2cf952c943c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:01.183679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37db3e1-f53a-43a2-9152-2cf952c943c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:01.185291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37db3e1-f53a-43a2-9152-2cf952c943c2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:01.195435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1013/2010] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:01.200453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48857b3e-39eb-4664-bcec-effc208b70e3 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:01.210742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48857b3e-39eb-4664-bcec-effc208b70e3 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:01.216230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:01.216893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:01.223092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:01.223092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] PATH_INFO: `/v3/projects/00f43d87dc7c49b8b85955c9f92878e8/users/918665feb7484ff39693adee44d2ec16/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:01.234518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00f43d87dc7c49b8b85955c9f92878e8, user_id=918665feb7484ff39693adee44d2ec16, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:01.237247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:01.340328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 998/2011] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:01 2022] PUT /identity/v3/projects/00f43d87dc7c49b8b85955c9f92878e8/users/918665feb7484ff39693adee44d2ec16/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:01.344784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4ea38b-1e1a-4684-8b18-6c13dd5bbe6b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:01.345084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4ea38b-1e1a-4684-8b18-6c13dd5bbe6b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:01.345359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4ea38b-1e1a-4684-8b18-6c13dd5bbe6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:01.380093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4e4ea38b-1e1a-4684-8b18-6c13dd5bbe6b None None] MFA Rules not processed for user `918665feb7484ff39693adee44d2ec16`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:01.381328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4e4ea38b-1e1a-4684-8b18-6c13dd5bbe6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:01.450189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1014/2012] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:01 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:01.463379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:01.466016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:01.569373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b9002c-3c93-4bd1-9ac9-64b31a510194 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:01.574765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a26598-d6d8-41fc-adca-3e2ed0784709 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:01.575513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a26598-d6d8-41fc-adca-3e2ed0784709 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:01.575860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a26598-d6d8-41fc-adca-3e2ed0784709 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:01.576205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a26598-d6d8-41fc-adca-3e2ed0784709 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:01.587875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a26598-d6d8-41fc-adca-3e2ed0784709 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:01.590302 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a26598-d6d8-41fc-adca-3e2ed0784709 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:01.602064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 999/2013] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:01 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:01.638547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:01.651185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4ea38b-1e1a-4684-8b18-6c13dd5bbe6b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:01.653345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4e4ea38b-1e1a-4684-8b18-6c13dd5bbe6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:01.733749 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4ea38b-1e1a-4684-8b18-6c13dd5bbe6b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:01.740018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:01.740956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:01.741667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:01.741667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] PATH_INFO: `/v3/projects/4e1adff6c1f74da19f8dab38ad35e5af` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:01.743380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e1adff6c1f74da19f8dab38ad35e5af)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:01.746591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:01.895597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e1adff6c1f74da19f8dab38ad35e5af', 'request_id': 'req-696092bb-5aa4-4d46-a43b-091dc35b723c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:01.901624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e1adff6c1f74da19f8dab38ad35e5af', 'request_id': 'req-696092bb-5aa4-4d46-a43b-091dc35b723c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.014823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e1adff6c1f74da19f8dab38ad35e5af', 'request_id': 'req-696092bb-5aa4-4d46-a43b-091dc35b723c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.039706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] The token cache is being invalidate because project 4e1adff6c1f74da19f8dab38ad35e5af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:15:02.040094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.040946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.042510 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1015/2014] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:15:01 2022] DELETE /identity/v3/projects/4e1adff6c1f74da19f8dab38ad35e5af => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:02.046519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a26598-d6d8-41fc-adca-3e2ed0784709 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:02.057677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a26598-d6d8-41fc-adca-3e2ed0784709 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:02.114819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:02.115471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:02.115775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:02.116047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] PATH_INFO: `/v3/users/73e8f158e9f446dba709cee4d528cb41` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:02.117241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=73e8f158e9f446dba709cee4d528cb41)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:02.119381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:02.254059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:02.399939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73e8f158e9f446dba709cee4d528cb41'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.411366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73e8f158e9f446dba709cee4d528cb41'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.412103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73e8f158e9f446dba709cee4d528cb41'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.533743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73e8f158e9f446dba709cee4d528cb41'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.541415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-80400cd4-a04d-4f80-81c6-4787561e4a1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73e8f158e9f446dba709cee4d528cb41'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:02.548359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1000/2015] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:15:02 2022] DELETE /identity/v3/users/73e8f158e9f446dba709cee4d528cb41 => generated 0 bytes in 502 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:02.797814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:03.054427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:03.056097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:03.076546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696092bb-5aa4-4d46-a43b-091dc35b723c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:03.139119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:03.139804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:03.140105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:03.140382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] PATH_INFO: `/v3/projects/cde899656eeb43828af519427bbcee68` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:03.141547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] RBAC: Authorizing `identity:delete_project(project_id=cde899656eeb43828af519427bbcee68)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:03.143624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:03.276503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cde899656eeb43828af519427bbcee68', 'request_id': 'req-d466b26f-8167-46c8-8edb-3b33df2c4cfc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:03.282730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cde899656eeb43828af519427bbcee68', 'request_id': 'req-d466b26f-8167-46c8-8edb-3b33df2c4cfc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:03.379309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cde899656eeb43828af519427bbcee68', 'request_id': 'req-d466b26f-8167-46c8-8edb-3b33df2c4cfc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:03.391277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] The token cache is being invalidate because project cde899656eeb43828af519427bbcee68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:15:03.391566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:03.392232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d466b26f-8167-46c8-8edb-3b33df2c4cfc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:03.393752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1016/2016] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:15:03 2022] DELETE /identity/v3/projects/cde899656eeb43828af519427bbcee68 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:03.399005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfea9f9-3dca-46af-a862-f4a11da5535b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:03.399290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfea9f9-3dca-46af-a862-f4a11da5535b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:03.399592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfea9f9-3dca-46af-a862-f4a11da5535b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:03.420703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-9cfea9f9-3dca-46af-a862-f4a11da5535b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:03.422003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9cfea9f9-3dca-46af-a862-f4a11da5535b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:03.479030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1001/2017] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:03.488275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f98000-c956-4662-8111-52e066021c57 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:03.488564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f98000-c956-4662-8111-52e066021c57 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:03.488841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f98000-c956-4662-8111-52e066021c57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:03.509502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f0f98000-c956-4662-8111-52e066021c57 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:03.510747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f0f98000-c956-4662-8111-52e066021c57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:03.518732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1017/2018] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:03.523427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfea9f9-3dca-46af-a862-f4a11da5535b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:03.532934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfea9f9-3dca-46af-a862-f4a11da5535b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:03.541079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:03.541703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:03.541988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:03.542257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:03.543434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:15:03.543732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:03.545315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:03.557559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1002/2019] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:15:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:15:03.562566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f98000-c956-4662-8111-52e066021c57 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:03.571990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f98000-c956-4662-8111-52e066021c57 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:03.576705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e0e8d-422a-4e94-89a2-b99c85bb5ad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:03.577302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e0e8d-422a-4e94-89a2-b99c85bb5ad8 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:03.577582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e0e8d-422a-4e94-89a2-b99c85bb5ad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:03.577927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e0e8d-422a-4e94-89a2-b99c85bb5ad8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:03.578566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99e0e8d-422a-4e94-89a2-b99c85bb5ad8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:03.580457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99e0e8d-422a-4e94-89a2-b99c85bb5ad8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:03.700925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1018/2020] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:15:03 2022] POST /identity/v3/projects => generated 375 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:15:03.705212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:03.715899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebbe993-e3ac-4b83-801b-fcf6507fd638 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:03.770831 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6383133-9b05-4390-9f54-a4f1a95c0c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:03.771436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6383133-9b05-4390-9f54-a4f1a95c0c78 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:03.771713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6383133-9b05-4390-9f54-a4f1a95c0c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:03.772017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6383133-9b05-4390-9f54-a4f1a95c0c78 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:03.772645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6383133-9b05-4390-9f54-a4f1a95c0c78 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:03.774606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6383133-9b05-4390-9f54-a4f1a95c0c78 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:03.959997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1003/2021] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:15:03 2022] POST /identity/v3/users => generated 288 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:15:03.961245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e0e8d-422a-4e94-89a2-b99c85bb5ad8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:03.974423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e0e8d-422a-4e94-89a2-b99c85bb5ad8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:03.979363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e548f543-49d4-419d-a3d7-16c8b06aa38d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:03.980048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e548f543-49d4-419d-a3d7-16c8b06aa38d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:03.980395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e548f543-49d4-419d-a3d7-16c8b06aa38d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:03.980732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e548f543-49d4-419d-a3d7-16c8b06aa38d admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:03.981448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e548f543-49d4-419d-a3d7-16c8b06aa38d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:03.983237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e548f543-49d4-419d-a3d7-16c8b06aa38d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:03.993519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1019/2022] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:03.998334 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6383133-9b05-4390-9f54-a4f1a95c0c78 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:04.008172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6383133-9b05-4390-9f54-a4f1a95c0c78 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:04.013311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79eeca18-6c3e-497f-b247-50a586691940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:04.013959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eeca18-6c3e-497f-b247-50a586691940 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.014238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eeca18-6c3e-497f-b247-50a586691940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.014513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79eeca18-6c3e-497f-b247-50a586691940 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.015155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79eeca18-6c3e-497f-b247-50a586691940 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:04.016752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79eeca18-6c3e-497f-b247-50a586691940 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:04.028112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1004/2023] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:04.032483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e548f543-49d4-419d-a3d7-16c8b06aa38d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:04.043164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e548f543-49d4-419d-a3d7-16c8b06aa38d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:04.047821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab0b0b7-d1d2-45ba-8a53-1b6ae57fbb44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:04.048538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0b0b7-d1d2-45ba-8a53-1b6ae57fbb44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.048884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0b0b7-d1d2-45ba-8a53-1b6ae57fbb44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.049240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab0b0b7-d1d2-45ba-8a53-1b6ae57fbb44 admin admin] PATH_INFO: `/v3/projects/5fa4f1c031e64e7ca1e96aeb517b14b1/users/d1f31010454947f58779af6fd7a30053/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.068246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab0b0b7-d1d2-45ba-8a53-1b6ae57fbb44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fa4f1c031e64e7ca1e96aeb517b14b1, user_id=d1f31010454947f58779af6fd7a30053, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:04.068246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab0b0b7-d1d2-45ba-8a53-1b6ae57fbb44 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:04.069616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:04.168507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1020/2024] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:04 2022] PUT /identity/v3/projects/5fa4f1c031e64e7ca1e96aeb517b14b1/users/d1f31010454947f58779af6fd7a30053/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:15:04.180011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b63c89-c0a6-44b6-9652-08eaf5118333 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.180245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b63c89-c0a6-44b6-9652-08eaf5118333 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.180522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b63c89-c0a6-44b6-9652-08eaf5118333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.208012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-60b63c89-c0a6-44b6-9652-08eaf5118333 None None] MFA Rules not processed for user `d1f31010454947f58779af6fd7a30053`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:04.211155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-60b63c89-c0a6-44b6-9652-08eaf5118333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:04.230147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:04.274536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1005/2025] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:04 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:04.279648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab0b0b7-d1d2-45ba-8a53-1b6ae57fbb44 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:04.343162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab0b0b7-d1d2-45ba-8a53-1b6ae57fbb44 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:04.348000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fcf44e-10b7-45be-9dd9-9252e8d96f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:04.348658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fcf44e-10b7-45be-9dd9-9252e8d96f33 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.348944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fcf44e-10b7-45be-9dd9-9252e8d96f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.349227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fcf44e-10b7-45be-9dd9-9252e8d96f33 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.349884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fcf44e-10b7-45be-9dd9-9252e8d96f33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:04.351804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fcf44e-10b7-45be-9dd9-9252e8d96f33 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:04.459116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1021/2026] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:15:04 2022] POST /identity/v3/projects => generated 375 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:15:04.460276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b63c89-c0a6-44b6-9652-08eaf5118333 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:04.471389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b63c89-c0a6-44b6-9652-08eaf5118333 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:04.528734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfe8320-9d0a-46b7-a916-9b7ef3a617f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:04.529358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe8320-9d0a-46b7-a916-9b7ef3a617f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.529677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe8320-9d0a-46b7-a916-9b7ef3a617f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.529975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfe8320-9d0a-46b7-a916-9b7ef3a617f3 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.530615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfe8320-9d0a-46b7-a916-9b7ef3a617f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:04.532549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfe8320-9d0a-46b7-a916-9b7ef3a617f3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:04.666002 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1006/2027] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:15:04 2022] POST /identity/v3/users => generated 294 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:15:04.672189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fcf44e-10b7-45be-9dd9-9252e8d96f33 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:04.682617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fcf44e-10b7-45be-9dd9-9252e8d96f33 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:04.687896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683d974-3d47-465d-ab9f-4a0240158e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:04.688522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683d974-3d47-465d-ab9f-4a0240158e96 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.688804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683d974-3d47-465d-ab9f-4a0240158e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.689082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683d974-3d47-465d-ab9f-4a0240158e96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.689909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1683d974-3d47-465d-ab9f-4a0240158e96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:04.691508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1683d974-3d47-465d-ab9f-4a0240158e96 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:04.702758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1022/2028] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:04.707021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfe8320-9d0a-46b7-a916-9b7ef3a617f3 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:04.717537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfe8320-9d0a-46b7-a916-9b7ef3a617f3 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:04.722366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa80613c-889f-4d74-b285-0d61f5fd2ed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:04.723024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa80613c-889f-4d74-b285-0d61f5fd2ed8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.723307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa80613c-889f-4d74-b285-0d61f5fd2ed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.723584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa80613c-889f-4d74-b285-0d61f5fd2ed8 admin admin] PATH_INFO: `/v3/projects/f38eac147071436aa439d4b0d6ef4122/users/e2394c9d7ed74ae2954609f8cc112c88/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.739286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa80613c-889f-4d74-b285-0d61f5fd2ed8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f38eac147071436aa439d4b0d6ef4122, user_id=e2394c9d7ed74ae2954609f8cc112c88, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:04.742065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa80613c-889f-4d74-b285-0d61f5fd2ed8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:04.837077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1007/2029] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:04 2022] PUT /identity/v3/projects/f38eac147071436aa439d4b0d6ef4122/users/e2394c9d7ed74ae2954609f8cc112c88/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:04.841428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683d974-3d47-465d-ab9f-4a0240158e96 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:04.851567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683d974-3d47-465d-ab9f-4a0240158e96 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:04.920521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705018eb-944c-43be-bb54-d249de05a72b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:04.921126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705018eb-944c-43be-bb54-d249de05a72b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.921404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705018eb-944c-43be-bb54-d249de05a72b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.921703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705018eb-944c-43be-bb54-d249de05a72b admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.922372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705018eb-944c-43be-bb54-d249de05a72b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:04.928150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705018eb-944c-43be-bb54-d249de05a72b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:04.942465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1023/2030] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:04 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:04.950293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa80613c-889f-4d74-b285-0d61f5fd2ed8 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:04.960624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa80613c-889f-4d74-b285-0d61f5fd2ed8 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:04.965754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:04.966412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.966772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.967047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] PATH_INFO: `/v3/projects/f38eac147071436aa439d4b0d6ef4122/users/e2394c9d7ed74ae2954609f8cc112c88/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:04.968954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f38eac147071436aa439d4b0d6ef4122, user_id=e2394c9d7ed74ae2954609f8cc112c88, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:04.971614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:04.982779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1008/2031] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:04 2022] PUT /identity/v3/projects/f38eac147071436aa439d4b0d6ef4122/users/e2394c9d7ed74ae2954609f8cc112c88/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:04.993118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af07f0-5278-46a1-9af6-7b54b2d10da0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:04.993414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af07f0-5278-46a1-9af6-7b54b2d10da0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:04.993720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39af07f0-5278-46a1-9af6-7b54b2d10da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:05.023844 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-39af07f0-5278-46a1-9af6-7b54b2d10da0 None None] MFA Rules not processed for user `e2394c9d7ed74ae2954609f8cc112c88`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:05.024920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-39af07f0-5278-46a1-9af6-7b54b2d10da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:05.075056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:05.095291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1024/2032] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:04 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:05.108367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:05.110239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:05.173962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cef0aef-b08a-46fc-aa2d-27860e7beb36 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:05.177674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afeb585-26a8-47b9-891b-d5ee4c45e585 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:05.178493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afeb585-26a8-47b9-891b-d5ee4c45e585 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:05.178794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afeb585-26a8-47b9-891b-d5ee4c45e585 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:05.179065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afeb585-26a8-47b9-891b-d5ee4c45e585 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:05.208377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:05.239260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afeb585-26a8-47b9-891b-d5ee4c45e585 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:05.241153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afeb585-26a8-47b9-891b-d5ee4c45e585 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:05.253719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1009/2033] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:05 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:09.591439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39af07f0-5278-46a1-9af6-7b54b2d10da0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:09.658067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39af07f0-5278-46a1-9af6-7b54b2d10da0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:09.663335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:09.663945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:09.664229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:09.664503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] PATH_INFO: `/v3/users/d1f31010454947f58779af6fd7a30053` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:09.665730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1f31010454947f58779af6fd7a30053)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:09.667664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:09.840107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1f31010454947f58779af6fd7a30053'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:09.840678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1f31010454947f58779af6fd7a30053'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:09.853390 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1f31010454947f58779af6fd7a30053'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:09.939551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1f31010454947f58779af6fd7a30053'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:09.947272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1f31010454947f58779af6fd7a30053'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:09.953964 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1025/2034] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:15:09 2022] DELETE /identity/v3/users/d1f31010454947f58779af6fd7a30053 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:15:09.958535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df999bdf-ea22-4998-88b9-1ed1ef6be28e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:09.959006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df999bdf-ea22-4998-88b9-1ed1ef6be28e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:09.959104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df999bdf-ea22-4998-88b9-1ed1ef6be28e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:09.978419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-df999bdf-ea22-4998-88b9-1ed1ef6be28e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:09.979643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-df999bdf-ea22-4998-88b9-1ed1ef6be28e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:10.041964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1010/2035] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:10.052352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:10.053993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:10.115103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38201c48-360a-481e-9f35-6f50901ea01c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:10.118699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e129e6-39c2-42b4-8d9d-6f248d6d3781 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:10.119356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e129e6-39c2-42b4-8d9d-6f248d6d3781 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:10.119657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e129e6-39c2-42b4-8d9d-6f248d6d3781 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:10.119990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e129e6-39c2-42b4-8d9d-6f248d6d3781 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:10.130612 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e129e6-39c2-42b4-8d9d-6f248d6d3781 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:10.132365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e129e6-39c2-42b4-8d9d-6f248d6d3781 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:10.142169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1026/2036] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:10.603900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df999bdf-ea22-4998-88b9-1ed1ef6be28e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:10.606103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-df999bdf-ea22-4998-88b9-1ed1ef6be28e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:10.623552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df999bdf-ea22-4998-88b9-1ed1ef6be28e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:10.628561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:10.629146 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:10.629521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:10.629861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] PATH_INFO: `/v3/projects/5fa4f1c031e64e7ca1e96aeb517b14b1` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:10.631042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] RBAC: Authorizing `identity:delete_project(project_id=5fa4f1c031e64e7ca1e96aeb517b14b1)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:10.634659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:10.777286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5fa4f1c031e64e7ca1e96aeb517b14b1', 'request_id': 'req-3990866b-0551-437d-8f55-60df86d21a9b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:10.899864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5fa4f1c031e64e7ca1e96aeb517b14b1', 'request_id': 'req-3990866b-0551-437d-8f55-60df86d21a9b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:10.908553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5fa4f1c031e64e7ca1e96aeb517b14b1', 'request_id': 'req-3990866b-0551-437d-8f55-60df86d21a9b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:10.920145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] The token cache is being invalidate because project 5fa4f1c031e64e7ca1e96aeb517b14b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:15:10.920435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:10.921113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:10.922592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1011/2037] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:15:10 2022] DELETE /identity/v3/projects/5fa4f1c031e64e7ca1e96aeb517b14b1 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:10.926722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e129e6-39c2-42b4-8d9d-6f248d6d3781 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:10.936612 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e129e6-39c2-42b4-8d9d-6f248d6d3781 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:10.944771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:10.997014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:10.997712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:10.997990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:10.998770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] PATH_INFO: `/v3/users/e2394c9d7ed74ae2954609f8cc112c88` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:10.999956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2394c9d7ed74ae2954609f8cc112c88)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:11.001907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:11.267505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2394c9d7ed74ae2954609f8cc112c88'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:11.268200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2394c9d7ed74ae2954609f8cc112c88'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:11.282642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2394c9d7ed74ae2954609f8cc112c88'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:11.431324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2394c9d7ed74ae2954609f8cc112c88'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:11.444539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-310b9a20-a729-41c8-8285-be49d25cc5e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2394c9d7ed74ae2954609f8cc112c88'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:11.453941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1027/2038] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:15:10 2022] DELETE /identity/v3/users/e2394c9d7ed74ae2954609f8cc112c88 => generated 0 bytes in 526 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:12.007822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:12.009314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:12.023484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3990866b-0551-437d-8f55-60df86d21a9b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:12.074872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:12.083501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:12.087280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:12.087280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:12.087280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] PATH_INFO: `/v3/projects/f38eac147071436aa439d4b0d6ef4122` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:12.087280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f38eac147071436aa439d4b0d6ef4122)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:12.087646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:12.239874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f38eac147071436aa439d4b0d6ef4122', 'request_id': 'req-53a8592c-20bb-4564-baed-8948e7025e41'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:12.328285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f38eac147071436aa439d4b0d6ef4122', 'request_id': 'req-53a8592c-20bb-4564-baed-8948e7025e41'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:12.339698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f38eac147071436aa439d4b0d6ef4122', 'request_id': 'req-53a8592c-20bb-4564-baed-8948e7025e41'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:12.348609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] The token cache is being invalidate because project f38eac147071436aa439d4b0d6ef4122 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:15:12.348968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:12.349672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-53a8592c-20bb-4564-baed-8948e7025e41 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:12.351344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1012/2039] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:15:12 2022] DELETE /identity/v3/projects/f38eac147071436aa439d4b0d6ef4122 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:15:12.357739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97fda3-1bad-44ad-8416-e7b60e9cf8ef None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:12.358230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97fda3-1bad-44ad-8416-e7b60e9cf8ef None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:12.358305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef97fda3-1bad-44ad-8416-e7b60e9cf8ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:12.387045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ef97fda3-1bad-44ad-8416-e7b60e9cf8ef None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:12.390064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ef97fda3-1bad-44ad-8416-e7b60e9cf8ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:12.492740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678d8b5e-57fe-4150-afcc-5c1523fc2e42 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:12.493027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678d8b5e-57fe-4150-afcc-5c1523fc2e42 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:12.493614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678d8b5e-57fe-4150-afcc-5c1523fc2e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:12.500020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1028/2040] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:12.520491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-678d8b5e-57fe-4150-afcc-5c1523fc2e42 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:12.523147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-678d8b5e-57fe-4150-afcc-5c1523fc2e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:12.535339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97fda3-1bad-44ad-8416-e7b60e9cf8ef None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:12.536174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1013/2041] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:15:12.548095 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef97fda3-1bad-44ad-8416-e7b60e9cf8ef None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:12.560101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:12.560936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:12.561207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:12.561673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:12.562552 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:15:12.563021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:12.565360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:12.580237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1029/2042] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:15:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:15:12.584333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678d8b5e-57fe-4150-afcc-5c1523fc2e42 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:12.595358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678d8b5e-57fe-4150-afcc-5c1523fc2e42 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:12.601189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a28d9-b02e-494b-93ff-234951e53a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:12.601923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a28d9-b02e-494b-93ff-234951e53a9d admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:12.602208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a28d9-b02e-494b-93ff-234951e53a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:12.602479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a28d9-b02e-494b-93ff-234951e53a9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:12.603117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841a28d9-b02e-494b-93ff-234951e53a9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:12.605660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841a28d9-b02e-494b-93ff-234951e53a9d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:12.746687 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1014/2043] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:15:12 2022] POST /identity/v3/projects => generated 373 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:15:12.750819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:12.760501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5109036-4adc-4da4-b191-01e86f565d81 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:12.772752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:12.819179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbb0d67-5c14-4943-8e5b-be2e5db49603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:12.819790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb0d67-5c14-4943-8e5b-be2e5db49603 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:12.820067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb0d67-5c14-4943-8e5b-be2e5db49603 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:12.820336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb0d67-5c14-4943-8e5b-be2e5db49603 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:12.820961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbb0d67-5c14-4943-8e5b-be2e5db49603 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:12.822891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbb0d67-5c14-4943-8e5b-be2e5db49603 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:12.927508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1030/2044] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:15:12 2022] POST /identity/v3/users => generated 287 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:15:12.929073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a28d9-b02e-494b-93ff-234951e53a9d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:12.945148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841a28d9-b02e-494b-93ff-234951e53a9d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:12.951119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6971d34-751c-463d-a085-732ff38cd6e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:12.951847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6971d34-751c-463d-a085-732ff38cd6e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:12.952398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6971d34-751c-463d-a085-732ff38cd6e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:12.952780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6971d34-751c-463d-a085-732ff38cd6e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:12.953520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6971d34-751c-463d-a085-732ff38cd6e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:12.955614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6971d34-751c-463d-a085-732ff38cd6e9 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:12.967329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1015/2045] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:12 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:15:12.971612 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbb0d67-5c14-4943-8e5b-be2e5db49603 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:12.981988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbb0d67-5c14-4943-8e5b-be2e5db49603 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:12.989193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c425a9c-5304-407c-a21c-70c422fdcb5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:12.990967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c425a9c-5304-407c-a21c-70c422fdcb5c admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:12.991470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c425a9c-5304-407c-a21c-70c422fdcb5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:12.992819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c425a9c-5304-407c-a21c-70c422fdcb5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:12.993702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c425a9c-5304-407c-a21c-70c422fdcb5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:12.995752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c425a9c-5304-407c-a21c-70c422fdcb5c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:13.006157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1031/2046] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:12 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:13.013668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6971d34-751c-463d-a085-732ff38cd6e9 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:13.023941 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6971d34-751c-463d-a085-732ff38cd6e9 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:13.029222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:13.030011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:13.039268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:13.039588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] PATH_INFO: `/v3/projects/bcd2ab4056374054994fc04593338ac4/users/88079e13618d4ca3a3f851ccbfe26c2d/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:13.068114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcd2ab4056374054994fc04593338ac4, user_id=88079e13618d4ca3a3f851ccbfe26c2d, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:13.084503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:13.222237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1016/2047] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:13 2022] PUT /identity/v3/projects/bcd2ab4056374054994fc04593338ac4/users/88079e13618d4ca3a3f851ccbfe26c2d/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:13.233796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a258dd8d-2f5d-4f40-be97-e96ef6a52ad5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:13.234367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a258dd8d-2f5d-4f40-be97-e96ef6a52ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:13.234783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a258dd8d-2f5d-4f40-be97-e96ef6a52ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:13.262768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a258dd8d-2f5d-4f40-be97-e96ef6a52ad5 None None] MFA Rules not processed for user `88079e13618d4ca3a3f851ccbfe26c2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:13.264361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a258dd8d-2f5d-4f40-be97-e96ef6a52ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:13.336798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1032/2048] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:13 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:13.348665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:13.350741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:13.410333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f890a5-97e4-4a28-98a1-4933a38d53b0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:13.415982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93513caf-fe7d-4dda-9924-9e2ce298add0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:13.415982 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93513caf-fe7d-4dda-9924-9e2ce298add0 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:13.416884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93513caf-fe7d-4dda-9924-9e2ce298add0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:13.417243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93513caf-fe7d-4dda-9924-9e2ce298add0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:13.427662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93513caf-fe7d-4dda-9924-9e2ce298add0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:13.429606 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93513caf-fe7d-4dda-9924-9e2ce298add0 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:13.446425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1017/2049] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:13 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:18.485232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8f3f6-433f-48f4-8e53-28120756b57e None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:18.485773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8f3f6-433f-48f4-8e53-28120756b57e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:18.485986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8f3f6-433f-48f4-8e53-28120756b57e None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:18.487558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1033/2050] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:15:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:15:18.516893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52d009c-432c-4a0c-9696-818dbe20e385 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:18.517406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52d009c-432c-4a0c-9696-818dbe20e385 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:18.517500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52d009c-432c-4a0c-9696-818dbe20e385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:18.539429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-b52d009c-432c-4a0c-9696-818dbe20e385 None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:18.540536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b52d009c-432c-4a0c-9696-818dbe20e385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:18.560559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:18.607536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1018/2051] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:15:18 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:18.622178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f8f3f6-433f-48f4-8e53-28120756b57e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:18.623907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f1f8f3f6-433f-48f4-8e53-28120756b57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:18.640480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f8f3f6-433f-48f4-8e53-28120756b57e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:18.646668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6258d000-38c7-4a52-8dcd-fb7aef67c36f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:18.647280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6258d000-38c7-4a52-8dcd-fb7aef67c36f service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:18.647561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6258d000-38c7-4a52-8dcd-fb7aef67c36f service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:18.650660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6258d000-38c7-4a52-8dcd-fb7aef67c36f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:18.652652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52d009c-432c-4a0c-9696-818dbe20e385 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:18.654068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b52d009c-432c-4a0c-9696-818dbe20e385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:18.659421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6258d000-38c7-4a52-8dcd-fb7aef67c36f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:18.661043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6258d000-38c7-4a52-8dcd-fb7aef67c36f service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:18.674143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1034/2052] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:18.731501 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52d009c-432c-4a0c-9696-818dbe20e385 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:18.736184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:18.736823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:18.737107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:18.737390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] PATH_INFO: `/v3/users/3107aef7eb954ac4845c8903b005ce93` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:18.738613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3107aef7eb954ac4845c8903b005ce93)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:18.740536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:18.935233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3107aef7eb954ac4845c8903b005ce93'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:18.948011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3107aef7eb954ac4845c8903b005ce93'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:18.949212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3107aef7eb954ac4845c8903b005ce93'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:19.053337 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3107aef7eb954ac4845c8903b005ce93'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:19.063229 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3db5b432-7d8a-41e7-839a-bcde093243f5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3107aef7eb954ac4845c8903b005ce93'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:19.067706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1019/2053] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:15:18 2022] DELETE /identity/v3/users/3107aef7eb954ac4845c8903b005ce93 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:19.313003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6258d000-38c7-4a52-8dcd-fb7aef67c36f service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:19.325460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6258d000-38c7-4a52-8dcd-fb7aef67c36f service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:19.383996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:19.384630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:19.384907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:19.385182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] PATH_INFO: `/v3/projects/e1f4463eb245468984f4747372d53fc6` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:19.386970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1f4463eb245468984f4747372d53fc6)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:19.388885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:19.404663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:19.539371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1f4463eb245468984f4747372d53fc6', 'request_id': 'req-a42b00b9-9ba8-4032-b0f2-430916cb8974'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:19.544823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1f4463eb245468984f4747372d53fc6', 'request_id': 'req-a42b00b9-9ba8-4032-b0f2-430916cb8974'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:19.674957 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1f4463eb245468984f4747372d53fc6', 'request_id': 'req-a42b00b9-9ba8-4032-b0f2-430916cb8974'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:19.686767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] The token cache is being invalidate because project e1f4463eb245468984f4747372d53fc6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:15:19.687056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:19.687679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a42b00b9-9ba8-4032-b0f2-430916cb8974 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:19.689153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1035/2054] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:15:19 2022] DELETE /identity/v3/projects/e1f4463eb245468984f4747372d53fc6 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:19.694150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fd4c3-676f-49a4-a2e2-0f34def517d9 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:19.694574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fd4c3-676f-49a4-a2e2-0f34def517d9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:19.694699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fd4c3-676f-49a4-a2e2-0f34def517d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:19.715685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-128fd4c3-676f-49a4-a2e2-0f34def517d9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:19.716815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-128fd4c3-676f-49a4-a2e2-0f34def517d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:19.751661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:19.789469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1020/2055] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:15:19.789937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890abae-70f9-42d6-a39a-2adc614274db None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:19.790240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890abae-70f9-42d6-a39a-2adc614274db None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:19.790547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a890abae-70f9-42d6-a39a-2adc614274db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:19.815521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a890abae-70f9-42d6-a39a-2adc614274db None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:19.816738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a890abae-70f9-42d6-a39a-2adc614274db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:19.826728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1036/2056] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:19.830540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128fd4c3-676f-49a4-a2e2-0f34def517d9 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:19.845704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128fd4c3-676f-49a4-a2e2-0f34def517d9 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:19.845704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:19.846621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:19.847134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:19.847586 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:19.848636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:15:19.849072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:19.850857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:19.863057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1021/2057] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:15:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:15:19.867312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a890abae-70f9-42d6-a39a-2adc614274db None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:19.878319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a890abae-70f9-42d6-a39a-2adc614274db None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:19.883114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52951bb2-0e2e-47f5-8a0b-869f5d0537f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:19.883836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52951bb2-0e2e-47f5-8a0b-869f5d0537f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:19.884118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52951bb2-0e2e-47f5-8a0b-869f5d0537f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:19.884387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52951bb2-0e2e-47f5-8a0b-869f5d0537f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:19.885008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52951bb2-0e2e-47f5-8a0b-869f5d0537f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:19.886969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52951bb2-0e2e-47f5-8a0b-869f5d0537f8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:19.969982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1037/2058] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:15:19 2022] POST /identity/v3/projects => generated 375 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:15:19.974001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:19.984516 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082f09eb-1df7-46b1-bf77-3a9a2c15b5ce admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:20.038833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990ca064-0cb2-4042-b161-682fb1c088a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:20.039437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ca064-0cb2-4042-b161-682fb1c088a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:20.039719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ca064-0cb2-4042-b161-682fb1c088a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:20.039989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ca064-0cb2-4042-b161-682fb1c088a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:20.040599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990ca064-0cb2-4042-b161-682fb1c088a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:20.043099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990ca064-0cb2-4042-b161-682fb1c088a6 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:20.157249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1022/2059] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:15:19 2022] POST /identity/v3/users => generated 288 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:15:20.161442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52951bb2-0e2e-47f5-8a0b-869f5d0537f8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:20.172400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52951bb2-0e2e-47f5-8a0b-869f5d0537f8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:20.177056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bd5ef3-dfbc-4d13-a16b-534531a3d58d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:20.177743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bd5ef3-dfbc-4d13-a16b-534531a3d58d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:20.178509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bd5ef3-dfbc-4d13-a16b-534531a3d58d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:20.178778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bd5ef3-dfbc-4d13-a16b-534531a3d58d admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:20.179418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bd5ef3-dfbc-4d13-a16b-534531a3d58d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:20.181020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bd5ef3-dfbc-4d13-a16b-534531a3d58d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:20.190860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1038/2060] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:15:20.194588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990ca064-0cb2-4042-b161-682fb1c088a6 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:20.204276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990ca064-0cb2-4042-b161-682fb1c088a6 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:20.208825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a9a8fc-e5de-469c-a917-b27e6318069a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:20.209422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9a8fc-e5de-469c-a917-b27e6318069a admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:20.209736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9a8fc-e5de-469c-a917-b27e6318069a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:20.210010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a9a8fc-e5de-469c-a917-b27e6318069a admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:20.210641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a9a8fc-e5de-469c-a917-b27e6318069a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:20.212217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a9a8fc-e5de-469c-a917-b27e6318069a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:20.229443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1023/2061] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:15:20.234075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bd5ef3-dfbc-4d13-a16b-534531a3d58d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:20.244003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bd5ef3-dfbc-4d13-a16b-534531a3d58d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:20.248615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:20.249346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:20.249722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:20.250056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] PATH_INFO: `/v3/projects/4987f6cd050f445b8912b388ab4e9221/users/268d55d57a864981a40baf4cacf182b5/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:20.264649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4987f6cd050f445b8912b388ab4e9221, user_id=268d55d57a864981a40baf4cacf182b5, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:20.267406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:20.340606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1039/2062] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:20 2022] PUT /identity/v3/projects/4987f6cd050f445b8912b388ab4e9221/users/268d55d57a864981a40baf4cacf182b5/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:20.351200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb6c84-5dfa-45d5-bcd0-62b1758793cc None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:20.351490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb6c84-5dfa-45d5-bcd0-62b1758793cc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:20.351765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb6c84-5dfa-45d5-bcd0-62b1758793cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:20.379401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-b8bb6c84-5dfa-45d5-bcd0-62b1758793cc None None] MFA Rules not processed for user `268d55d57a864981a40baf4cacf182b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:20.380590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b8bb6c84-5dfa-45d5-bcd0-62b1758793cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:20.445755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1024/2063] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:20 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:20.457024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:20.459517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:20.534662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16c7393-8a07-4a76-bd21-be274ac73f1a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:20.540263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a468ed5-3478-4bbe-ba16-b43cdd4fb8f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:20.540904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a468ed5-3478-4bbe-ba16-b43cdd4fb8f4 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:20.541184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a468ed5-3478-4bbe-ba16-b43cdd4fb8f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:20.541458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a468ed5-3478-4bbe-ba16-b43cdd4fb8f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:20.549096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a468ed5-3478-4bbe-ba16-b43cdd4fb8f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:20.550704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a468ed5-3478-4bbe-ba16-b43cdd4fb8f4 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:20.559382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1040/2064] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:20 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:20.758629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb6c84-5dfa-45d5-bcd0-62b1758793cc None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:20.792672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:20.820762 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb6c84-5dfa-45d5-bcd0-62b1758793cc None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:20.825810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:20.826604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:20.826884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:20.827157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] PATH_INFO: `/v3/users/268d55d57a864981a40baf4cacf182b5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:20.828265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=268d55d57a864981a40baf4cacf182b5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:20.830311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:21.079584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '268d55d57a864981a40baf4cacf182b5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.092523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '268d55d57a864981a40baf4cacf182b5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.093214 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '268d55d57a864981a40baf4cacf182b5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.225138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '268d55d57a864981a40baf4cacf182b5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.232733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '268d55d57a864981a40baf4cacf182b5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.239006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1025/2065] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:15:20 2022] DELETE /identity/v3/users/268d55d57a864981a40baf4cacf182b5 => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:21.243357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66880225-ddd2-448d-9a80-3035ae17fffd None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:21.243644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66880225-ddd2-448d-9a80-3035ae17fffd None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:21.243921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66880225-ddd2-448d-9a80-3035ae17fffd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:21.264803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-66880225-ddd2-448d-9a80-3035ae17fffd None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:21.264803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-66880225-ddd2-448d-9a80-3035ae17fffd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:21.284858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:21.326681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1041/2066] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:21.336554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:21.338054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:21.403421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6838f1-4451-4299-bda2-63bb183fcb13 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:21.405422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:21.406175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:21.406620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:21.406990 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:21.418419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:21.420183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:21.431204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1026/2067] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:21.561244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66880225-ddd2-448d-9a80-3035ae17fffd None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:21.562878 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-66880225-ddd2-448d-9a80-3035ae17fffd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:21.575285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66880225-ddd2-448d-9a80-3035ae17fffd None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:21.580176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:21.581800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:21.581800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:21.581800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] PATH_INFO: `/v3/users/88079e13618d4ca3a3f851ccbfe26c2d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:21.582520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] RBAC: Authorizing `identity:delete_user(user_id=88079e13618d4ca3a3f851ccbfe26c2d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:21.584571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:21.772069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88079e13618d4ca3a3f851ccbfe26c2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.772726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88079e13618d4ca3a3f851ccbfe26c2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.786110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88079e13618d4ca3a3f851ccbfe26c2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.876749 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88079e13618d4ca3a3f851ccbfe26c2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.884544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:21.886629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:21.887650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b507045c-3c6b-4174-bb95-59c52f51b18b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88079e13618d4ca3a3f851ccbfe26c2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:21.896442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1042/2068] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:15:21 2022] DELETE /identity/v3/users/88079e13618d4ca3a3f851ccbfe26c2d => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:21.901163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac95fac4-2812-4983-99d0-d932adcb615f service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:21.902215 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36f5ad-403f-4a29-9f49-7971709bd785 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:21.902575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36f5ad-403f-4a29-9f49-7971709bd785 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:21.902931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d36f5ad-403f-4a29-9f49-7971709bd785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:21.925863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3d36f5ad-403f-4a29-9f49-7971709bd785 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:21.927099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3d36f5ad-403f-4a29-9f49-7971709bd785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:21.965960 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:21.974923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:21.975648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:21.975997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:21.976340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] PATH_INFO: `/v3/projects/4987f6cd050f445b8912b388ab4e9221` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:21.978062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4987f6cd050f445b8912b388ab4e9221)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:21.980795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:22.004691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1043/2069] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:22.025363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d36f5ad-403f-4a29-9f49-7971709bd785 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:22.025363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3d36f5ad-403f-4a29-9f49-7971709bd785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:22.061980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:22.076965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d36f5ad-403f-4a29-9f49-7971709bd785 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:22.080911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f25197-ee72-43ee-985e-cf074a48175a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:22.081482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f25197-ee72-43ee-985e-cf074a48175a service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:22.081805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f25197-ee72-43ee-985e-cf074a48175a service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:22.108920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f25197-ee72-43ee-985e-cf074a48175a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:22.108920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f25197-ee72-43ee-985e-cf074a48175a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:22.108920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f25197-ee72-43ee-985e-cf074a48175a service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:22.108920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1044/2070] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:22.193741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4987f6cd050f445b8912b388ab4e9221', 'request_id': 'req-3d7db33d-1329-467c-af10-77a2138d29e2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:22.320792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4987f6cd050f445b8912b388ab4e9221', 'request_id': 'req-3d7db33d-1329-467c-af10-77a2138d29e2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:22.326806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4987f6cd050f445b8912b388ab4e9221', 'request_id': 'req-3d7db33d-1329-467c-af10-77a2138d29e2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:22.338551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] The token cache is being invalidate because project 4987f6cd050f445b8912b388ab4e9221 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:15:22.339497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:22.340430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3d7db33d-1329-467c-af10-77a2138d29e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:22.342191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1027/2071] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:15:21 2022] DELETE /identity/v3/projects/4987f6cd050f445b8912b388ab4e9221 => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:15:22.347127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e7c03-ba65-401b-894a-625b522b456c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:22.347419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e7c03-ba65-401b-894a-625b522b456c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:22.347760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e7c03-ba65-401b-894a-625b522b456c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:22.376908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-359e7c03-ba65-401b-894a-625b522b456c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:22.378274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-359e7c03-ba65-401b-894a-625b522b456c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:22.501646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1045/2072] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:22.504265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df6c3c-5d9a-477d-8126-53980f593af8 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:22.504551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df6c3c-5d9a-477d-8126-53980f593af8 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:22.504821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df6c3c-5d9a-477d-8126-53980f593af8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:22.533145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f1df6c3c-5d9a-477d-8126-53980f593af8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:22.534996 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f1df6c3c-5d9a-477d-8126-53980f593af8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:22.544534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1028/2073] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:22.548939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e7c03-ba65-401b-894a-625b522b456c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:22.559143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e7c03-ba65-401b-894a-625b522b456c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:22.564440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:22.568998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:22.568998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:22.568998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:22.568998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:15:22.568998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:22.568998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:22.581123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1046/2074] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:15:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:15:22.586092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1df6c3c-5d9a-477d-8126-53980f593af8 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:22.596647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1df6c3c-5d9a-477d-8126-53980f593af8 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:22.601647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85579fcf-5c4b-459e-95b3-331848673e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:22.602261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85579fcf-5c4b-459e-95b3-331848673e96 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:22.602567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85579fcf-5c4b-459e-95b3-331848673e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:22.602862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85579fcf-5c4b-459e-95b3-331848673e96 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:22.603487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85579fcf-5c4b-459e-95b3-331848673e96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:22.606062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85579fcf-5c4b-459e-95b3-331848673e96 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:22.799448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:22.805072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:22.811155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1029/2075] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:15:22 2022] POST /identity/v3/projects => generated 371 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:15:22.816211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85579fcf-5c4b-459e-95b3-331848673e96 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:22.833383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d7e067-522f-49d9-abf5-c6bb6c2c0439 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:22.891833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85579fcf-5c4b-459e-95b3-331848673e96 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:22.896714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0874d2-24e8-4834-acac-097a86333025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:22.899075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0874d2-24e8-4834-acac-097a86333025 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:22.899075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0874d2-24e8-4834-acac-097a86333025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:22.899075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0874d2-24e8-4834-acac-097a86333025 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:22.899075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0874d2-24e8-4834-acac-097a86333025 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:22.900407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0874d2-24e8-4834-acac-097a86333025 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:22.904019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:22.904641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:22.904916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:22.905183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] PATH_INFO: `/v3/projects/bcd2ab4056374054994fc04593338ac4` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:22.906314 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcd2ab4056374054994fc04593338ac4)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:22.908236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:23.067543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1030/2076] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:15:22 2022] POST /identity/v3/users => generated 286 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:15:23.076944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0874d2-24e8-4834-acac-097a86333025 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:23.086959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0874d2-24e8-4834-acac-097a86333025 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:23.091798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155f965f-7d1d-42c7-aa43-6ba9cef2a906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:23.092405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155f965f-7d1d-42c7-aa43-6ba9cef2a906 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:23.092697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155f965f-7d1d-42c7-aa43-6ba9cef2a906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:23.092965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155f965f-7d1d-42c7-aa43-6ba9cef2a906 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:23.093661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155f965f-7d1d-42c7-aa43-6ba9cef2a906 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:23.095361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155f965f-7d1d-42c7-aa43-6ba9cef2a906 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:23.108125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1031/2077] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:15:23.111770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155f965f-7d1d-42c7-aa43-6ba9cef2a906 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:23.117933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcd2ab4056374054994fc04593338ac4', 'request_id': 'req-0a44b605-8e42-4668-8841-79f8ebc8c0f6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:23.122029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155f965f-7d1d-42c7-aa43-6ba9cef2a906 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:23.124069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcd2ab4056374054994fc04593338ac4', 'request_id': 'req-0a44b605-8e42-4668-8841-79f8ebc8c0f6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:23.179189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba367ce2-89cd-4c14-9a89-ac2b12aa2354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:23.179801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba367ce2-89cd-4c14-9a89-ac2b12aa2354 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:23.180082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba367ce2-89cd-4c14-9a89-ac2b12aa2354 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:23.180383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba367ce2-89cd-4c14-9a89-ac2b12aa2354 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:23.181342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba367ce2-89cd-4c14-9a89-ac2b12aa2354 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:23.183559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba367ce2-89cd-4c14-9a89-ac2b12aa2354 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:23.193125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1032/2078] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:23 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:15:23.197383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba367ce2-89cd-4c14-9a89-ac2b12aa2354 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:23.207700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba367ce2-89cd-4c14-9a89-ac2b12aa2354 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:23.212400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:23.213032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:23.213308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:23.213577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] PATH_INFO: `/v3/projects/2868eaafdeb84f779a6a923a694248b9/users/0c84d85f5ada4422a1f5895bfdb33025/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:23.228436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2868eaafdeb84f779a6a923a694248b9, user_id=0c84d85f5ada4422a1f5895bfdb33025, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:23.231109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:23.237152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcd2ab4056374054994fc04593338ac4', 'request_id': 'req-0a44b605-8e42-4668-8841-79f8ebc8c0f6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:23.254006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] The token cache is being invalidate because project bcd2ab4056374054994fc04593338ac4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:15:23.254310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:23.255140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a44b605-8e42-4668-8841-79f8ebc8c0f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:15:23.256873 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1047/2079] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:15:22 2022] DELETE /identity/v3/projects/bcd2ab4056374054994fc04593338ac4 => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:23.357272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1033/2080] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:23 2022] PUT /identity/v3/projects/2868eaafdeb84f779a6a923a694248b9/users/0c84d85f5ada4422a1f5895bfdb33025/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:15:23.358914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc5284-cf44-4c6b-94d8-3fa956d62762 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:23.359231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc5284-cf44-4c6b-94d8-3fa956d62762 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:23.359508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc5284-cf44-4c6b-94d8-3fa956d62762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:23.394407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b5dc5284-cf44-4c6b-94d8-3fa956d62762 None None] MFA Rules not processed for user `0c84d85f5ada4422a1f5895bfdb33025`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:23.395861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b5dc5284-cf44-4c6b-94d8-3fa956d62762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:23.478504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1048/2081] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:23 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:15:23.483527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:23.485291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:23.564157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38482c99-8c88-42c8-9d74-5f5bd0a7e214 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:23.568908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52658702-d967-4ffe-a96e-ed143fb18674 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:23.569510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52658702-d967-4ffe-a96e-ed143fb18674 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:23.569840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52658702-d967-4ffe-a96e-ed143fb18674 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:23.570113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52658702-d967-4ffe-a96e-ed143fb18674 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:23.570728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52658702-d967-4ffe-a96e-ed143fb18674 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:23.572734 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52658702-d967-4ffe-a96e-ed143fb18674 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:23.757500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:23.766575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1034/2082] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:15:23 2022] POST /identity/v3/projects => generated 371 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:15:23.770723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dc5284-cf44-4c6b-94d8-3fa956d62762 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:23.781800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dc5284-cf44-4c6b-94d8-3fa956d62762 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:23.837664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db945e3b-0d87-4657-a1fd-83b8a06e1885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:23.838269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db945e3b-0d87-4657-a1fd-83b8a06e1885 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:23.838542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db945e3b-0d87-4657-a1fd-83b8a06e1885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:23.839319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db945e3b-0d87-4657-a1fd-83b8a06e1885 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:23.839926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db945e3b-0d87-4657-a1fd-83b8a06e1885 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:23.842252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db945e3b-0d87-4657-a1fd-83b8a06e1885 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:23.952733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1049/2083] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:15:23 2022] POST /identity/v3/users => generated 292 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:15:23.956736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52658702-d967-4ffe-a96e-ed143fb18674 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:23.968714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52658702-d967-4ffe-a96e-ed143fb18674 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:23.973744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f601155-d617-4098-aa41-b5be4c4a55a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:23.974493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f601155-d617-4098-aa41-b5be4c4a55a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:23.974843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f601155-d617-4098-aa41-b5be4c4a55a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:23.975313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f601155-d617-4098-aa41-b5be4c4a55a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:23.976080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f601155-d617-4098-aa41-b5be4c4a55a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:23.977814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f601155-d617-4098-aa41-b5be4c4a55a8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:23.990580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1035/2084] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:23.995241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db945e3b-0d87-4657-a1fd-83b8a06e1885 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:24.005783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db945e3b-0d87-4657-a1fd-83b8a06e1885 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:24.010475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b67f0c-6bc1-449e-ae50-44b31eed7f75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:24.011234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b67f0c-6bc1-449e-ae50-44b31eed7f75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:24.011686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b67f0c-6bc1-449e-ae50-44b31eed7f75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:24.011958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b67f0c-6bc1-449e-ae50-44b31eed7f75 admin admin] PATH_INFO: `/v3/projects/c86860f3fc0a420f8e59a527df904d32/users/6bf72ffd4d6e4cf193e46c3963eb096b/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:24.030214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b67f0c-6bc1-449e-ae50-44b31eed7f75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c86860f3fc0a420f8e59a527df904d32, user_id=6bf72ffd4d6e4cf193e46c3963eb096b, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:24.033638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b67f0c-6bc1-449e-ae50-44b31eed7f75 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:24.035869 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:24.191162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1050/2085] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:23 2022] PUT /identity/v3/projects/c86860f3fc0a420f8e59a527df904d32/users/6bf72ffd4d6e4cf193e46c3963eb096b/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:24.194726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f601155-d617-4098-aa41-b5be4c4a55a8 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:24.205149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f601155-d617-4098-aa41-b5be4c4a55a8 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:24.262897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3483cd-317b-4717-8744-151440b498d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:24.263505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3483cd-317b-4717-8744-151440b498d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:24.263782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3483cd-317b-4717-8744-151440b498d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:24.264051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3483cd-317b-4717-8744-151440b498d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:24.264679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3483cd-317b-4717-8744-151440b498d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:24.266303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3483cd-317b-4717-8744-151440b498d9 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:24.276110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1036/2086] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:15:24.281060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b67f0c-6bc1-449e-ae50-44b31eed7f75 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:24.291306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b67f0c-6bc1-449e-ae50-44b31eed7f75 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:24.295860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:24.296499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:24.296775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:24.297259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] PATH_INFO: `/v3/projects/c86860f3fc0a420f8e59a527df904d32/users/6bf72ffd4d6e4cf193e46c3963eb096b/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:24.299303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c86860f3fc0a420f8e59a527df904d32, user_id=6bf72ffd4d6e4cf193e46c3963eb096b, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:24.302056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:24.315603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1051/2087] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:15:24 2022] PUT /identity/v3/projects/c86860f3fc0a420f8e59a527df904d32/users/6bf72ffd4d6e4cf193e46c3963eb096b/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:15:24.327845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d0f44-5c1b-4d63-bfdb-f325142d5bb6 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:24.328137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d0f44-5c1b-4d63-bfdb-f325142d5bb6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:24.328420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d0f44-5c1b-4d63-bfdb-f325142d5bb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:24.372683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ac8d0f44-5c1b-4d63-bfdb-f325142d5bb6 None None] MFA Rules not processed for user `6bf72ffd4d6e4cf193e46c3963eb096b`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:24.372683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ac8d0f44-5c1b-4d63-bfdb-f325142d5bb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:24.437195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1037/2088] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:24 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:15:24.450959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:24.452724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:24.511159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f931e7-9081-40a8-9523-4ad1730ce327 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:24.514890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a0bf12-1aa8-4f14-a22a-04dffbe131ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:24.515500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a0bf12-1aa8-4f14-a22a-04dffbe131ff service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:24.515778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a0bf12-1aa8-4f14-a22a-04dffbe131ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:24.516051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a0bf12-1aa8-4f14-a22a-04dffbe131ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:24.525935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a0bf12-1aa8-4f14-a22a-04dffbe131ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:24.533095 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a0bf12-1aa8-4f14-a22a-04dffbe131ff service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:24.537549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1052/2089] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:24 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:24.792772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d0f44-5c1b-4d63-bfdb-f325142d5bb6 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:15:24.802187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d0f44-5c1b-4d63-bfdb-f325142d5bb6 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:15:24.805815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10eb6c3a-8b02-47eb-8518-1b67d7b5863f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:15:24.806433 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb6c3a-8b02-47eb-8518-1b67d7b5863f service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:24.806724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb6c3a-8b02-47eb-8518-1b67d7b5863f service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:24.806995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10eb6c3a-8b02-47eb-8518-1b67d7b5863f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:24.863539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eb6c3a-8b02-47eb-8518-1b67d7b5863f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:15:24.864981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10eb6c3a-8b02-47eb-8518-1b67d7b5863f service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:15:24.874384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1038/2090] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:15:24 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:15:25.238348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc83cb0-4ba9-4072-a5b1-e4f939b96452 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:25.238944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc83cb0-4ba9-4072-a5b1-e4f939b96452 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:25.238944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc83cb0-4ba9-4072-a5b1-e4f939b96452 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:25.259149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-dbc83cb0-4ba9-4072-a5b1-e4f939b96452 None None] MFA Rules not processed for user `5be22eb85b3b4828a36edc6e1fcc05f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:25.260333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-dbc83cb0-4ba9-4072-a5b1-e4f939b96452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:25.312451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1053/2091] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:15:25.431375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:25.482896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:36.008391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=54 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:36.122103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=33 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:15:57.885158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0279182-3450-407a-91be-c433173d3ccc None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:15:57.885714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0279182-3450-407a-91be-c433173d3ccc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:15:57.885826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0279182-3450-407a-91be-c433173d3ccc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:15:57.905084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f0279182-3450-407a-91be-c433173d3ccc None None] MFA Rules not processed for user `b6ca9962541744bf9d67cc3e05721413`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:15:57.906309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f0279182-3450-407a-91be-c433173d3ccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:15:57.961178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1039/2092] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:15:57 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:03.305903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f6bf7-0034-4dbe-bc40-d39f2c8ed6c6 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:03.312025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f6bf7-0034-4dbe-bc40-d39f2c8ed6c6 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:03.312025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f6bf7-0034-4dbe-bc40-d39f2c8ed6c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:03.326212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-100f6bf7-0034-4dbe-bc40-d39f2c8ed6c6 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:03.327408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-100f6bf7-0034-4dbe-bc40-d39f2c8ed6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:03.388040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1054/2093] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:16:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:03.432786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0279182-3450-407a-91be-c433173d3ccc None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:03.434287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f0279182-3450-407a-91be-c433173d3ccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:03.506044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0279182-3450-407a-91be-c433173d3ccc None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:03.513086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48bdc07-ed02-4160-a3ae-34d24820ab92 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:03.514298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bdc07-ed02-4160-a3ae-34d24820ab92 service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:03.514903 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bdc07-ed02-4160-a3ae-34d24820ab92 service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:03.515191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48bdc07-ed02-4160-a3ae-34d24820ab92 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:03.523867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48bdc07-ed02-4160-a3ae-34d24820ab92 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:03.525483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48bdc07-ed02-4160-a3ae-34d24820ab92 service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:03.533670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1040/2094] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:16:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:16:13.379337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=25 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:13.530355 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=57 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:15.992450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100f6bf7-0034-4dbe-bc40-d39f2c8ed6c6 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:15.994035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-100f6bf7-0034-4dbe-bc40-d39f2c8ed6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:16.123327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100f6bf7-0034-4dbe-bc40-d39f2c8ed6c6 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:16.129235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:16.130071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:16.130365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:16.130979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] PATH_INFO: `/v3/users/918665feb7484ff39693adee44d2ec16` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:16.132528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] RBAC: Authorizing `identity:delete_user(user_id=918665feb7484ff39693adee44d2ec16)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:16.134769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:16.383978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '918665feb7484ff39693adee44d2ec16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:16.386005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '918665feb7484ff39693adee44d2ec16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:16.428166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '918665feb7484ff39693adee44d2ec16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:16.664243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '918665feb7484ff39693adee44d2ec16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:16.673493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '918665feb7484ff39693adee44d2ec16'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:16.687513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1055/2095] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:15 2022] DELETE /identity/v3/users/918665feb7484ff39693adee44d2ec16 => generated 0 bytes in 688 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:16.687649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3272ba3-f5f0-4a7e-a527-cc5a29017203 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:16.687649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3272ba3-f5f0-4a7e-a527-cc5a29017203 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:16.687649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3272ba3-f5f0-4a7e-a527-cc5a29017203 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:16.707035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e3272ba3-f5f0-4a7e-a527-cc5a29017203 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:16.708662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e3272ba3-f5f0-4a7e-a527-cc5a29017203 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:16.781627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1041/2096] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:16.794159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:16.796004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:16.887852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a64c3c-ff01-4eec-8c4c-665bfa6f501f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:16.892068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c27c8e-27d9-4829-9c5b-1ab1d9081cc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:16.892933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c27c8e-27d9-4829-9c5b-1ab1d9081cc0 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:16.893309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c27c8e-27d9-4829-9c5b-1ab1d9081cc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:16.893731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c27c8e-27d9-4829-9c5b-1ab1d9081cc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:16.905251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c27c8e-27d9-4829-9c5b-1ab1d9081cc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:16.906930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c27c8e-27d9-4829-9c5b-1ab1d9081cc0 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:16.919742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1056/2097] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:16:17.450088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3272ba3-f5f0-4a7e-a527-cc5a29017203 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:17.454032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e3272ba3-f5f0-4a7e-a527-cc5a29017203 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:17.477618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3272ba3-f5f0-4a7e-a527-cc5a29017203 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:17.492029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:17.492523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:17.493007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:17.493482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] PATH_INFO: `/v3/projects/00f43d87dc7c49b8b85955c9f92878e8` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:17.495645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=00f43d87dc7c49b8b85955c9f92878e8)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:17.497958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:17.664138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00f43d87dc7c49b8b85955c9f92878e8', 'request_id': 'req-b46fe114-4512-4ef4-94f2-3a70f85a83dd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:17.790699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00f43d87dc7c49b8b85955c9f92878e8', 'request_id': 'req-b46fe114-4512-4ef4-94f2-3a70f85a83dd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:17.796820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00f43d87dc7c49b8b85955c9f92878e8', 'request_id': 'req-b46fe114-4512-4ef4-94f2-3a70f85a83dd'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:17.808674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] The token cache is being invalidate because project 00f43d87dc7c49b8b85955c9f92878e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:17.808962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:17.809619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b46fe114-4512-4ef4-94f2-3a70f85a83dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:17.811111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1042/2098] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:17 2022] DELETE /identity/v3/projects/00f43d87dc7c49b8b85955c9f92878e8 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:17.816105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0948431-619b-47ff-a352-ebbd7808a2bf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:17.824098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0948431-619b-47ff-a352-ebbd7808a2bf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:17.824098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0948431-619b-47ff-a352-ebbd7808a2bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:17.836814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e0948431-619b-47ff-a352-ebbd7808a2bf None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:17.838058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e0948431-619b-47ff-a352-ebbd7808a2bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:17.855361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:17.898679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1057/2099] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:17.906871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96295f1-727e-42f1-af84-9c61a739f23c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:17.907279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96295f1-727e-42f1-af84-9c61a739f23c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:17.908519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96295f1-727e-42f1-af84-9c61a739f23c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:17.931369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f96295f1-727e-42f1-af84-9c61a739f23c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:17.933220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f96295f1-727e-42f1-af84-9c61a739f23c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:17.941665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1043/2100] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:17.946015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0948431-619b-47ff-a352-ebbd7808a2bf None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:17.957639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0948431-619b-47ff-a352-ebbd7808a2bf None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:17.964074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:17.964684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:17.964962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:17.965234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:17.966021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:16:17.966317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:17.970921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:17.987380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1058/2101] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:16:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:16:17.988642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96295f1-727e-42f1-af84-9c61a739f23c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:18.002226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96295f1-727e-42f1-af84-9c61a739f23c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:18.005059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e42873-a3d4-4219-81f5-c93978a2516c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:18.005720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e42873-a3d4-4219-81f5-c93978a2516c admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:18.006000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e42873-a3d4-4219-81f5-c93978a2516c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:18.006278 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e42873-a3d4-4219-81f5-c93978a2516c admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:18.006909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e42873-a3d4-4219-81f5-c93978a2516c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:18.008867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e42873-a3d4-4219-81f5-c93978a2516c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:18.212286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1044/2102] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:17 2022] POST /identity/v3/projects => generated 373 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:16:18.213385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:18.224442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28db940f-2209-45c2-b676-fe45886a430b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:18.318741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a21cae5-bf30-4beb-ad3f-7d82f53abdb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:18.319368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a21cae5-bf30-4beb-ad3f-7d82f53abdb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:18.319648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a21cae5-bf30-4beb-ad3f-7d82f53abdb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:18.319918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a21cae5-bf30-4beb-ad3f-7d82f53abdb1 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:18.320560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a21cae5-bf30-4beb-ad3f-7d82f53abdb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:18.323079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a21cae5-bf30-4beb-ad3f-7d82f53abdb1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:18.446335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1059/2103] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:18 2022] POST /identity/v3/users => generated 287 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:16:18.453679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e42873-a3d4-4219-81f5-c93978a2516c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:18.462089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e42873-a3d4-4219-81f5-c93978a2516c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:18.488725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484c6199-8f93-4012-b80e-b99c71e42cc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:18.489658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484c6199-8f93-4012-b80e-b99c71e42cc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:18.489948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484c6199-8f93-4012-b80e-b99c71e42cc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:18.490226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484c6199-8f93-4012-b80e-b99c71e42cc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:18.490955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484c6199-8f93-4012-b80e-b99c71e42cc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:18.495023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484c6199-8f93-4012-b80e-b99c71e42cc0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:18.508335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1045/2104] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:18 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:18.512262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a21cae5-bf30-4beb-ad3f-7d82f53abdb1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:18.527447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a21cae5-bf30-4beb-ad3f-7d82f53abdb1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:18.532914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b1bd93-6c31-4db8-bdee-d5ea5b019da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:18.533640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b1bd93-6c31-4db8-bdee-d5ea5b019da2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:18.534109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b1bd93-6c31-4db8-bdee-d5ea5b019da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:18.534455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b1bd93-6c31-4db8-bdee-d5ea5b019da2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:18.535174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b1bd93-6c31-4db8-bdee-d5ea5b019da2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:18.536850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b1bd93-6c31-4db8-bdee-d5ea5b019da2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:18.551986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1060/2105] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:18 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:18.564990 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484c6199-8f93-4012-b80e-b99c71e42cc0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:18.588228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484c6199-8f93-4012-b80e-b99c71e42cc0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:18.595000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:18.596204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:18.596631 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:18.596985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] PATH_INFO: `/v3/projects/b56bcc90049a4312b6ab84df3f6cd64d/users/c5dfa5e282f3436cbb126600a14e8cfb/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:18.614712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b56bcc90049a4312b6ab84df3f6cd64d, user_id=c5dfa5e282f3436cbb126600a14e8cfb, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:18.618068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:18.740117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1046/2106] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:18 2022] PUT /identity/v3/projects/b56bcc90049a4312b6ab84df3f6cd64d/users/c5dfa5e282f3436cbb126600a14e8cfb/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:18.756438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d96e5a-fb4d-4887-ac57-a3e4d5172297 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:18.757159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d96e5a-fb4d-4887-ac57-a3e4d5172297 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:18.757437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d96e5a-fb4d-4887-ac57-a3e4d5172297 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:18.815364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-02d96e5a-fb4d-4887-ac57-a3e4d5172297 None None] MFA Rules not processed for user `c5dfa5e282f3436cbb126600a14e8cfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:18.818290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-02d96e5a-fb4d-4887-ac57-a3e4d5172297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:18.897241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1061/2107] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:18 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:18.909774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:18.918263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:18.977531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:19.027910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ca5496-03fd-4a35-8100-a2d24875bb44 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:19.032211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b916af3-d4a7-472b-b855-686b71c5bdda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:19.032830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b916af3-d4a7-472b-b855-686b71c5bdda service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:19.033103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b916af3-d4a7-472b-b855-686b71c5bdda service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:19.033370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b916af3-d4a7-472b-b855-686b71c5bdda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:19.044010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b916af3-d4a7-472b-b855-686b71c5bdda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:19.045744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b916af3-d4a7-472b-b855-686b71c5bdda service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:19.057491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1047/2108] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:19.088150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d96e5a-fb4d-4887-ac57-a3e4d5172297 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:19.109861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:19.162769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d96e5a-fb4d-4887-ac57-a3e4d5172297 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:19.168343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:19.169040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:19.169326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:19.169620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] PATH_INFO: `/v3/users/c5dfa5e282f3436cbb126600a14e8cfb` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:19.170948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5dfa5e282f3436cbb126600a14e8cfb)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:19.172992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:19.395267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5dfa5e282f3436cbb126600a14e8cfb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:19.395973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5dfa5e282f3436cbb126600a14e8cfb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:19.411214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5dfa5e282f3436cbb126600a14e8cfb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:19.527724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5dfa5e282f3436cbb126600a14e8cfb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:19.536433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5dfa5e282f3436cbb126600a14e8cfb'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:19.543836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1062/2109] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:19 2022] DELETE /identity/v3/users/c5dfa5e282f3436cbb126600a14e8cfb => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:19.548503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd09dfc-1d3f-48cc-9484-764bebdf8ad5 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:19.548899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd09dfc-1d3f-48cc-9484-764bebdf8ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:19.549282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd09dfc-1d3f-48cc-9484-764bebdf8ad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:19.568813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2dd09dfc-1d3f-48cc-9484-764bebdf8ad5 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:19.570175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2dd09dfc-1d3f-48cc-9484-764bebdf8ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:19.641981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1048/2110] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:19.652450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:19.654176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:19.734889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51841cdb-4c74-4429-ab52-a1ad66a33760 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:19.739171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228c1726-bde7-4df1-ba69-b6fe9747cc21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:19.740001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c1726-bde7-4df1-ba69-b6fe9747cc21 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:19.740450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c1726-bde7-4df1-ba69-b6fe9747cc21 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:19.740839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c1726-bde7-4df1-ba69-b6fe9747cc21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:19.751411 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228c1726-bde7-4df1-ba69-b6fe9747cc21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:19.752909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228c1726-bde7-4df1-ba69-b6fe9747cc21 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:19.769142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1063/2111] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:20.325504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd09dfc-1d3f-48cc-9484-764bebdf8ad5 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:20.327321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2dd09dfc-1d3f-48cc-9484-764bebdf8ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:20.347845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd09dfc-1d3f-48cc-9484-764bebdf8ad5 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:20.354208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:20.355387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:20.355387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:20.355593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] PATH_INFO: `/v3/projects/b56bcc90049a4312b6ab84df3f6cd64d` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:20.357116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] RBAC: Authorizing `identity:delete_project(project_id=b56bcc90049a4312b6ab84df3f6cd64d)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:20.359956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:20.520776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b56bcc90049a4312b6ab84df3f6cd64d', 'request_id': 'req-b569cb15-bb57-4a2a-b720-4113d3a13aac'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:20.607939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b56bcc90049a4312b6ab84df3f6cd64d', 'request_id': 'req-b569cb15-bb57-4a2a-b720-4113d3a13aac'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:20.616268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b56bcc90049a4312b6ab84df3f6cd64d', 'request_id': 'req-b569cb15-bb57-4a2a-b720-4113d3a13aac'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:20.631252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] The token cache is being invalidate because project b56bcc90049a4312b6ab84df3f6cd64d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:20.632874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:20.632874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b569cb15-bb57-4a2a-b720-4113d3a13aac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:20.634093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1049/2112] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:20 2022] DELETE /identity/v3/projects/b56bcc90049a4312b6ab84df3f6cd64d => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:20.639198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6c9b13-738d-417e-b88b-f7db875d31b3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:20.639659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6c9b13-738d-417e-b88b-f7db875d31b3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:20.639750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6c9b13-738d-417e-b88b-f7db875d31b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:20.659749 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8b6c9b13-738d-417e-b88b-f7db875d31b3 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:20.660956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8b6c9b13-738d-417e-b88b-f7db875d31b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:20.705423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:20.741236 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1064/2113] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:20.750927 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eab60f3-7c67-4f0e-a8a5-5b40f0a7ca8f None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:20.751248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eab60f3-7c67-4f0e-a8a5-5b40f0a7ca8f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:20.751680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eab60f3-7c67-4f0e-a8a5-5b40f0a7ca8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:20.773978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2eab60f3-7c67-4f0e-a8a5-5b40f0a7ca8f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:20.775103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2eab60f3-7c67-4f0e-a8a5-5b40f0a7ca8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:20.788568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1050/2114] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:16:20.796083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6c9b13-738d-417e-b88b-f7db875d31b3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:20.805998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6c9b13-738d-417e-b88b-f7db875d31b3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:20.811468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:20.812093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:20.812374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:20.812663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:20.813323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:16:20.813647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:20.815279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:20.849219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1065/2115] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:16:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:16:20.853571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eab60f3-7c67-4f0e-a8a5-5b40f0a7ca8f None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:20.864566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eab60f3-7c67-4f0e-a8a5-5b40f0a7ca8f None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:20.873782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592503ec-5880-4d1a-91ab-8e2664153998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:20.874407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592503ec-5880-4d1a-91ab-8e2664153998 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:20.874683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592503ec-5880-4d1a-91ab-8e2664153998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:20.874949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592503ec-5880-4d1a-91ab-8e2664153998 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:20.875580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592503ec-5880-4d1a-91ab-8e2664153998 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:20.878348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592503ec-5880-4d1a-91ab-8e2664153998 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:21.018352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1051/2116] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:20 2022] POST /identity/v3/projects => generated 375 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:16:21.019890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:21.031433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96448b3f-10ec-4d47-bca8-68547778db97 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:21.118243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9a88be-35be-4718-b77e-10e39f611e62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:21.118853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9a88be-35be-4718-b77e-10e39f611e62 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:21.119155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9a88be-35be-4718-b77e-10e39f611e62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:21.119425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9a88be-35be-4718-b77e-10e39f611e62 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:21.120052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9a88be-35be-4718-b77e-10e39f611e62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:21.122069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9a88be-35be-4718-b77e-10e39f611e62 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:21.213920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1066/2117] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:21 2022] POST /identity/v3/users => generated 288 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:21.215186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592503ec-5880-4d1a-91ab-8e2664153998 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:21.226635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592503ec-5880-4d1a-91ab-8e2664153998 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:21.233899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044609f4-eaac-4a43-bf24-29fc3668923f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:21.234519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044609f4-eaac-4a43-bf24-29fc3668923f admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:21.234817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044609f4-eaac-4a43-bf24-29fc3668923f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:21.235174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044609f4-eaac-4a43-bf24-29fc3668923f admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:21.235853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044609f4-eaac-4a43-bf24-29fc3668923f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:21.237518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044609f4-eaac-4a43-bf24-29fc3668923f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:21.248250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1052/2118] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:21 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:21.252270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9a88be-35be-4718-b77e-10e39f611e62 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:21.265467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9a88be-35be-4718-b77e-10e39f611e62 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:21.271106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10586ace-b30e-4b9d-b330-a631e3e839d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:21.271771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10586ace-b30e-4b9d-b330-a631e3e839d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:21.272055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10586ace-b30e-4b9d-b330-a631e3e839d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:21.272361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10586ace-b30e-4b9d-b330-a631e3e839d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:21.273009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10586ace-b30e-4b9d-b330-a631e3e839d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:21.280238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10586ace-b30e-4b9d-b330-a631e3e839d0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:21.298971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044609f4-eaac-4a43-bf24-29fc3668923f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:21.300578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1067/2119] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:21 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:21.312368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044609f4-eaac-4a43-bf24-29fc3668923f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:21.319994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:21.320725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:21.321004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:21.321279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] PATH_INFO: `/v3/projects/1ba0701fef4648c8b434d92a5588afd9/users/b9cb460463b44f89b391e145746190c0/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:21.326642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:21.343811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ba0701fef4648c8b434d92a5588afd9, user_id=b9cb460463b44f89b391e145746190c0, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:21.346887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:21.422129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1053/2120] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:21 2022] PUT /identity/v3/projects/1ba0701fef4648c8b434d92a5588afd9/users/b9cb460463b44f89b391e145746190c0/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:21.432888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856780e-45c3-424b-824a-94fd4dbc1c83 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:21.433329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856780e-45c3-424b-824a-94fd4dbc1c83 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:21.433882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856780e-45c3-424b-824a-94fd4dbc1c83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:21.471557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8856780e-45c3-424b-824a-94fd4dbc1c83 None None] MFA Rules not processed for user `b9cb460463b44f89b391e145746190c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:21.472820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8856780e-45c3-424b-824a-94fd4dbc1c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:21.564910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1068/2121] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:21.580504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:21.583186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:21.696423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e49bbed-12d9-479e-bb89-c9f1195dad52 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:21.701298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ad0d0e-09a4-449b-9ad3-b84a1736cb9e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:21.701945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad0d0e-09a4-449b-9ad3-b84a1736cb9e service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:21.702223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad0d0e-09a4-449b-9ad3-b84a1736cb9e service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:21.702484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad0d0e-09a4-449b-9ad3-b84a1736cb9e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:21.713797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ad0d0e-09a4-449b-9ad3-b84a1736cb9e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:21.715553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ad0d0e-09a4-449b-9ad3-b84a1736cb9e service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:21.733314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1054/2122] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:21 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:24.713934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238535b-f62e-4b08-864a-55eda135a087 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:24.714347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238535b-f62e-4b08-864a-55eda135a087 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:24.714529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a238535b-f62e-4b08-864a-55eda135a087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:24.746030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a238535b-f62e-4b08-864a-55eda135a087 None None] MFA Rules not processed for user `b9cb460463b44f89b391e145746190c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:24.747537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a238535b-f62e-4b08-864a-55eda135a087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:24.760678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1069/2123] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:24 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:26.879836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090fb7c3-0f04-4902-81c8-292d7f0f4354 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:26.880317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090fb7c3-0f04-4902-81c8-292d7f0f4354 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:26.880424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090fb7c3-0f04-4902-81c8-292d7f0f4354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:26.899605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-090fb7c3-0f04-4902-81c8-292d7f0f4354 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:26.900806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-090fb7c3-0f04-4902-81c8-292d7f0f4354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:26.955074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1055/2124] 10.222.0.16 () {58 vars in 967 bytes} [Thu Aug 25 05:16:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:26.968249 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a238535b-f62e-4b08-864a-55eda135a087 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:26.969673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a238535b-f62e-4b08-864a-55eda135a087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:27.000820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:27.037937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a238535b-f62e-4b08-864a-55eda135a087 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:27.042950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1239e366-b11d-42a5-974c-f39b7d83fe9e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:27.043554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239e366-b11d-42a5-974c-f39b7d83fe9e service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:27.043835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239e366-b11d-42a5-974c-f39b7d83fe9e service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:27.044104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239e366-b11d-42a5-974c-f39b7d83fe9e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:27.052193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1239e366-b11d-42a5-974c-f39b7d83fe9e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:27.053809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1239e366-b11d-42a5-974c-f39b7d83fe9e service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:27.062212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1070/2125] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:16:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:16:33.752920 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10821f2-ba70-4db3-828a-9fed000602a3 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:33.753888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10821f2-ba70-4db3-828a-9fed000602a3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:33.754284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10821f2-ba70-4db3-828a-9fed000602a3 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:33.755767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1056/2126] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:16:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:16:33.790883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195fbaa5-2535-4290-a6c2-1b3bb879b99f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:33.791504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195fbaa5-2535-4290-a6c2-1b3bb879b99f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:33.792164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195fbaa5-2535-4290-a6c2-1b3bb879b99f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:33.815801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-195fbaa5-2535-4290-a6c2-1b3bb879b99f None None] MFA Rules not processed for user `e1ffa92d7fb546399bc404053c201426`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:33.817019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-195fbaa5-2535-4290-a6c2-1b3bb879b99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:33.890378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1071/2127] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:16:33 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:33.898126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10821f2-ba70-4db3-828a-9fed000602a3 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:33.902453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f10821f2-ba70-4db3-828a-9fed000602a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:33.917581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10821f2-ba70-4db3-828a-9fed000602a3 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:33.922819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9673a69e-8f60-48cc-a6dd-833683b92f52 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:33.923377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9673a69e-8f60-48cc-a6dd-833683b92f52 service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:33.923685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9673a69e-8f60-48cc-a6dd-833683b92f52 service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:33.924128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9673a69e-8f60-48cc-a6dd-833683b92f52 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:33.934478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9673a69e-8f60-48cc-a6dd-833683b92f52 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:33.935882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9673a69e-8f60-48cc-a6dd-833683b92f52 service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:33.941686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195fbaa5-2535-4290-a6c2-1b3bb879b99f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:33.943748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-195fbaa5-2535-4290-a6c2-1b3bb879b99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:33.949124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1057/2128] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:34.019830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195fbaa5-2535-4290-a6c2-1b3bb879b99f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:34.024668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:34.025831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:34.026828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:34.027118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] PATH_INFO: `/v3/users/5be22eb85b3b4828a36edc6e1fcc05f7` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:34.028307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5be22eb85b3b4828a36edc6e1fcc05f7)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:34.030385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:34.217356 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5be22eb85b3b4828a36edc6e1fcc05f7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:34.220595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5be22eb85b3b4828a36edc6e1fcc05f7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:34.228716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:34.235135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5be22eb85b3b4828a36edc6e1fcc05f7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:34.356468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5be22eb85b3b4828a36edc6e1fcc05f7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:34.364894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6f203aa-da6d-418d-8b39-52b6312a3181 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5be22eb85b3b4828a36edc6e1fcc05f7'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:34.375430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1072/2129] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:33 2022] DELETE /identity/v3/users/5be22eb85b3b4828a36edc6e1fcc05f7 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:34.706382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9673a69e-8f60-48cc-a6dd-833683b92f52 service placement] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:34.717499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9673a69e-8f60-48cc-a6dd-833683b92f52 service placement] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:34.797388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:34.798147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:34.799046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:34.802340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] PATH_INFO: `/v3/projects/a252c620742a4f6b9fdaafb4cb81f10f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:34.802340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a252c620742a4f6b9fdaafb4cb81f10f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:34.805700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:34.838356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:34.951227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a252c620742a4f6b9fdaafb4cb81f10f', 'request_id': 'req-b0bf6609-6930-406f-9ca5-b26f565e8949'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:35.065031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a252c620742a4f6b9fdaafb4cb81f10f', 'request_id': 'req-b0bf6609-6930-406f-9ca5-b26f565e8949'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:35.071812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a252c620742a4f6b9fdaafb4cb81f10f', 'request_id': 'req-b0bf6609-6930-406f-9ca5-b26f565e8949'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:35.084477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] The token cache is being invalidate because project a252c620742a4f6b9fdaafb4cb81f10f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:35.084774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:35.085426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0bf6609-6930-406f-9ca5-b26f565e8949 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:35.087342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1058/2130] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:34 2022] DELETE /identity/v3/projects/a252c620742a4f6b9fdaafb4cb81f10f => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:35.093124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f450bbc3-17e0-4117-bd53-9e9fea2b16d1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:35.093575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f450bbc3-17e0-4117-bd53-9e9fea2b16d1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:35.093722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f450bbc3-17e0-4117-bd53-9e9fea2b16d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:35.115975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f450bbc3-17e0-4117-bd53-9e9fea2b16d1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:35.117261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f450bbc3-17e0-4117-bd53-9e9fea2b16d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:35.190645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1073/2131] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:35.199096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05d1e0-6564-408a-89e3-731026adbb0d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:35.199494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05d1e0-6564-408a-89e3-731026adbb0d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:35.199766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05d1e0-6564-408a-89e3-731026adbb0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:35.232362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1e05d1e0-6564-408a-89e3-731026adbb0d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:35.233489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1e05d1e0-6564-408a-89e3-731026adbb0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:35.242704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1059/2132] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:35.246809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f450bbc3-17e0-4117-bd53-9e9fea2b16d1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:35.256374 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f450bbc3-17e0-4117-bd53-9e9fea2b16d1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:35.442209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:35.442209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:35.442209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:35.442209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:35.443109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:16:35.443109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:35.443109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:35.443109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1074/2133] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:16:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:16:35.467702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e05d1e0-6564-408a-89e3-731026adbb0d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:35.478580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e05d1e0-6564-408a-89e3-731026adbb0d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:35.483850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbf6168-10a6-4972-897b-0516f30301bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:35.493870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf6168-10a6-4972-897b-0516f30301bb admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:35.493870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf6168-10a6-4972-897b-0516f30301bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:35.493870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf6168-10a6-4972-897b-0516f30301bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:35.493870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbf6168-10a6-4972-897b-0516f30301bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:35.493870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbf6168-10a6-4972-897b-0516f30301bb admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:35.638713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1060/2134] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:35 2022] POST /identity/v3/projects => generated 399 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:35.642983 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:35.655968 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6249e4-7f5a-48c3-a3fe-7aa20843d47c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:35.721238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29d26c0-9370-4ff0-8774-f368bad37c80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:35.721805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d26c0-9370-4ff0-8774-f368bad37c80 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:35.722158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d26c0-9370-4ff0-8774-f368bad37c80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:35.722432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d26c0-9370-4ff0-8774-f368bad37c80 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:35.723067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29d26c0-9370-4ff0-8774-f368bad37c80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:35.724979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29d26c0-9370-4ff0-8774-f368bad37c80 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:35.877885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1075/2135] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:35 2022] POST /identity/v3/users => generated 300 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:35.882060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbf6168-10a6-4972-897b-0516f30301bb admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:35.892723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbf6168-10a6-4972-897b-0516f30301bb admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:35.897187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6927f30-440e-4025-b5a2-186bb94359f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:35.897980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6927f30-440e-4025-b5a2-186bb94359f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:35.898307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6927f30-440e-4025-b5a2-186bb94359f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:35.898530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6927f30-440e-4025-b5a2-186bb94359f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:35.899164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6927f30-440e-4025-b5a2-186bb94359f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:35.901256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6927f30-440e-4025-b5a2-186bb94359f0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:35.912027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1061/2136] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:35.916238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29d26c0-9370-4ff0-8774-f368bad37c80 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:35.926881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29d26c0-9370-4ff0-8774-f368bad37c80 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:35.936413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbbc2dd-e017-4907-b29b-0b4e234e3dbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:35.936413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbbc2dd-e017-4907-b29b-0b4e234e3dbd admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:35.936413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbbc2dd-e017-4907-b29b-0b4e234e3dbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:35.936413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbbc2dd-e017-4907-b29b-0b4e234e3dbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:35.936797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbbc2dd-e017-4907-b29b-0b4e234e3dbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:35.936797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbbc2dd-e017-4907-b29b-0b4e234e3dbd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:35.945861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1076/2137] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:35.949878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6927f30-440e-4025-b5a2-186bb94359f0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:35.959894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6927f30-440e-4025-b5a2-186bb94359f0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:35.964934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:35.965582 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:35.965893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:35.966170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] PATH_INFO: `/v3/projects/a134dff9974748abbd1d559a7f822708/users/deae76d7994f40a69b4041fdfdf499d8/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:35.986180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a134dff9974748abbd1d559a7f822708, user_id=deae76d7994f40a69b4041fdfdf499d8, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:35.986180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:36.093560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1062/2138] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:35 2022] PUT /identity/v3/projects/a134dff9974748abbd1d559a7f822708/users/deae76d7994f40a69b4041fdfdf499d8/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:36.103699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f9bf9a-fb6d-4d4a-9f27-6488021014f5 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:36.104210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f9bf9a-fb6d-4d4a-9f27-6488021014f5 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:36.104313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f9bf9a-fb6d-4d4a-9f27-6488021014f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:36.132149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b9f9bf9a-fb6d-4d4a-9f27-6488021014f5 None None] MFA Rules not processed for user `deae76d7994f40a69b4041fdfdf499d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:36.133348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b9f9bf9a-fb6d-4d4a-9f27-6488021014f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:36.162635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:36.197888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1077/2139] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:36 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:36.209128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:36.210981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:36.270444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf72e4b-be84-4463-8811-9df6ebf4b455 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:36.274045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f7e7ef-3972-4e8d-b3c9-7a1d4c7ad19e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:36.276749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f7e7ef-3972-4e8d-b3c9-7a1d4c7ad19e service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:36.276749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f7e7ef-3972-4e8d-b3c9-7a1d4c7ad19e service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:36.276749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f7e7ef-3972-4e8d-b3c9-7a1d4c7ad19e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:36.283680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f7e7ef-3972-4e8d-b3c9-7a1d4c7ad19e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:36.285102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f7e7ef-3972-4e8d-b3c9-7a1d4c7ad19e service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:36.294642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1063/2140] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:36 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:37.685493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f9bf9a-fb6d-4d4a-9f27-6488021014f5 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:37.686949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b9f9bf9a-fb6d-4d4a-9f27-6488021014f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:37.752441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f9bf9a-fb6d-4d4a-9f27-6488021014f5 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:37.756922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:37.757545 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:37.757849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:37.758117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] PATH_INFO: `/v3/users/0c84d85f5ada4422a1f5895bfdb33025` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:37.759223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c84d85f5ada4422a1f5895bfdb33025)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:37.761758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:37.952804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c84d85f5ada4422a1f5895bfdb33025'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:37.953547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c84d85f5ada4422a1f5895bfdb33025'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:37.965768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c84d85f5ada4422a1f5895bfdb33025'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:38.064820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c84d85f5ada4422a1f5895bfdb33025'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:38.072416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c84d85f5ada4422a1f5895bfdb33025'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:38.078733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1078/2141] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:37 2022] DELETE /identity/v3/users/0c84d85f5ada4422a1f5895bfdb33025 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:38.083108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90bcfc4-470b-4324-8435-9149b96e3d81 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:38.083708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90bcfc4-470b-4324-8435-9149b96e3d81 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:38.083708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90bcfc4-470b-4324-8435-9149b96e3d81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:38.103273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e90bcfc4-470b-4324-8435-9149b96e3d81 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:38.105071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e90bcfc4-470b-4324-8435-9149b96e3d81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:38.146245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:38.163293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1064/2142] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:38.173496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:38.174994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:38.230900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:38.234885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bf244e-7c6d-41ec-a742-66708da07d0e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:38.238856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea6f5a0-ae2c-4c7a-9c86-74e2b0c20be5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:38.239471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6f5a0-ae2c-4c7a-9c86-74e2b0c20be5 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:38.239926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6f5a0-ae2c-4c7a-9c86-74e2b0c20be5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:38.240200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea6f5a0-ae2c-4c7a-9c86-74e2b0c20be5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:38.249335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea6f5a0-ae2c-4c7a-9c86-74e2b0c20be5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:38.250822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea6f5a0-ae2c-4c7a-9c86-74e2b0c20be5 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:38.260007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1079/2143] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:38.729648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90bcfc4-470b-4324-8435-9149b96e3d81 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:38.731232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e90bcfc4-470b-4324-8435-9149b96e3d81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:38.743183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90bcfc4-470b-4324-8435-9149b96e3d81 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:38.748487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:38.749125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:38.749407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:38.749724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] PATH_INFO: `/v3/projects/2868eaafdeb84f779a6a923a694248b9` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:38.750839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2868eaafdeb84f779a6a923a694248b9)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:38.752768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:38.881515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2868eaafdeb84f779a6a923a694248b9', 'request_id': 'req-764d962c-36c2-42d5-96b6-723430046471'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:38.973320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2868eaafdeb84f779a6a923a694248b9', 'request_id': 'req-764d962c-36c2-42d5-96b6-723430046471'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:38.979824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2868eaafdeb84f779a6a923a694248b9', 'request_id': 'req-764d962c-36c2-42d5-96b6-723430046471'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:38.994768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] The token cache is being invalidate because project 2868eaafdeb84f779a6a923a694248b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:38.994768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:38.994768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:38.994768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1065/2144] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:38 2022] DELETE /identity/v3/projects/2868eaafdeb84f779a6a923a694248b9 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:38.997209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea6f5a0-ae2c-4c7a-9c86-74e2b0c20be5 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:39.006735 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea6f5a0-ae2c-4c7a-9c86-74e2b0c20be5 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:39.062842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:39.063560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:39.063837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:39.064106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] PATH_INFO: `/v3/users/6bf72ffd4d6e4cf193e46c3963eb096b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:39.065211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bf72ffd4d6e4cf193e46c3963eb096b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:39.067176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:39.279394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bf72ffd4d6e4cf193e46c3963eb096b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:39.280007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bf72ffd4d6e4cf193e46c3963eb096b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:39.291899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bf72ffd4d6e4cf193e46c3963eb096b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:39.378489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bf72ffd4d6e4cf193e46c3963eb096b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:39.386180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e26589ce-333b-4361-aba9-33fe7126e047 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bf72ffd4d6e4cf193e46c3963eb096b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:39.392431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1080/2145] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:38 2022] DELETE /identity/v3/users/6bf72ffd4d6e4cf193e46c3963eb096b => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:39.838799 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:39.840293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:39.856942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d962c-36c2-42d5-96b6-723430046471 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:39.914605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:39.915244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:39.915526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:39.915794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] PATH_INFO: `/v3/projects/c86860f3fc0a420f8e59a527df904d32` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:39.917499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] RBAC: Authorizing `identity:delete_project(project_id=c86860f3fc0a420f8e59a527df904d32)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:39.919720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:40.031381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c86860f3fc0a420f8e59a527df904d32', 'request_id': 'req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:40.161896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c86860f3fc0a420f8e59a527df904d32', 'request_id': 'req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:40.168046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c86860f3fc0a420f8e59a527df904d32', 'request_id': 'req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:40.175854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:40.179602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] The token cache is being invalidate because project c86860f3fc0a420f8e59a527df904d32 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:40.179892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:40.180519 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-72cdaa1a-742f-4e0a-b715-3f7dbdf4538f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:40.182039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1066/2146] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:39 2022] DELETE /identity/v3/projects/c86860f3fc0a420f8e59a527df904d32 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:40.187658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeb7cdf-2ecd-4801-9c64-5a81d25b42ef None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.188130 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeb7cdf-2ecd-4801-9c64-5a81d25b42ef None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.188577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeb7cdf-2ecd-4801-9c64-5a81d25b42ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.208338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6aeb7cdf-2ecd-4801-9c64-5a81d25b42ef None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:40.215002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6aeb7cdf-2ecd-4801-9c64-5a81d25b42ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:40.289316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1081/2147] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:40.296989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd30d64-666c-4c85-b3a2-8f9a4dd1784e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.297274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd30d64-666c-4c85-b3a2-8f9a4dd1784e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.297545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd30d64-666c-4c85-b3a2-8f9a4dd1784e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.320439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-9cd30d64-666c-4c85-b3a2-8f9a4dd1784e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:40.321816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9cd30d64-666c-4c85-b3a2-8f9a4dd1784e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:40.332696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1067/2148] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:40.338115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeb7cdf-2ecd-4801-9c64-5a81d25b42ef None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:40.347973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeb7cdf-2ecd-4801-9c64-5a81d25b42ef None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:40.353015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:40.353858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.354754 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.355097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.355837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:16:40.356213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:40.358602 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:40.367835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:40.373910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1082/2149] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:16:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:16:40.381676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd30d64-666c-4c85-b3a2-8f9a4dd1784e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:40.391907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd30d64-666c-4c85-b3a2-8f9a4dd1784e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:40.397216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e47f94-376e-4dd7-9cac-c569c3460142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:40.397910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e47f94-376e-4dd7-9cac-c569c3460142 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.398194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e47f94-376e-4dd7-9cac-c569c3460142 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.398466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e47f94-376e-4dd7-9cac-c569c3460142 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.399110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e47f94-376e-4dd7-9cac-c569c3460142 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:40.401747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e47f94-376e-4dd7-9cac-c569c3460142 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:40.535180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1068/2150] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:40 2022] POST /identity/v3/projects => generated 369 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:40.539195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:40.549694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e3262f-4d34-4f0e-ab62-c5c42be97fb8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:40.606033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039f1f12-3939-49bb-8878-ca7357ad0695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:40.606662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f1f12-3939-49bb-8878-ca7357ad0695 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.606943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f1f12-3939-49bb-8878-ca7357ad0695 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.607218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039f1f12-3939-49bb-8878-ca7357ad0695 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.607839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039f1f12-3939-49bb-8878-ca7357ad0695 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:40.609779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039f1f12-3939-49bb-8878-ca7357ad0695 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:40.757073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1083/2151] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:40 2022] POST /identity/v3/users => generated 285 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:40.761161 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e47f94-376e-4dd7-9cac-c569c3460142 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:40.771838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e47f94-376e-4dd7-9cac-c569c3460142 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:40.776419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416e15b-7189-4eb2-94e0-35e3db8ef85f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:40.777019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416e15b-7189-4eb2-94e0-35e3db8ef85f admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.777303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416e15b-7189-4eb2-94e0-35e3db8ef85f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.777579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416e15b-7189-4eb2-94e0-35e3db8ef85f admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.778271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7416e15b-7189-4eb2-94e0-35e3db8ef85f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:40.779862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7416e15b-7189-4eb2-94e0-35e3db8ef85f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:40.789729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1069/2152] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:40.794060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039f1f12-3939-49bb-8878-ca7357ad0695 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:40.804105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039f1f12-3939-49bb-8878-ca7357ad0695 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:40.809319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7062dcb-53fb-4fae-ac06-00262d70f87f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:40.809951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7062dcb-53fb-4fae-ac06-00262d70f87f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.810234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7062dcb-53fb-4fae-ac06-00262d70f87f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.810508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7062dcb-53fb-4fae-ac06-00262d70f87f admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.811190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7062dcb-53fb-4fae-ac06-00262d70f87f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:40.812773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7062dcb-53fb-4fae-ac06-00262d70f87f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:40.823271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1084/2153] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:40.827504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416e15b-7189-4eb2-94e0-35e3db8ef85f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:40.837333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416e15b-7189-4eb2-94e0-35e3db8ef85f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:40.841837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c860c656-0ef7-40df-abd6-57053c59a06e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:40.842746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c860c656-0ef7-40df-abd6-57053c59a06e admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.843025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c860c656-0ef7-40df-abd6-57053c59a06e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.843296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c860c656-0ef7-40df-abd6-57053c59a06e admin admin] PATH_INFO: `/v3/projects/50d92682bc5f4346ba53e5bb43ca210f/users/dec8c83eb6ea41edb22b57d77397e71b/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.858127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c860c656-0ef7-40df-abd6-57053c59a06e admin admin] RBAC: Authorizing `identity:create_grant(project_id=50d92682bc5f4346ba53e5bb43ca210f, user_id=dec8c83eb6ea41edb22b57d77397e71b, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:40.860755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c860c656-0ef7-40df-abd6-57053c59a06e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:40.944384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1070/2154] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:40 2022] PUT /identity/v3/projects/50d92682bc5f4346ba53e5bb43ca210f/users/dec8c83eb6ea41edb22b57d77397e71b/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:40.956416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f43638-f0e4-45b6-8919-216cb3676b00 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:40.956700 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f43638-f0e4-45b6-8919-216cb3676b00 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:40.956966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f43638-f0e4-45b6-8919-216cb3676b00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:40.984785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-71f43638-f0e4-45b6-8919-216cb3676b00 None None] MFA Rules not processed for user `dec8c83eb6ea41edb22b57d77397e71b`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:40.986036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-71f43638-f0e4-45b6-8919-216cb3676b00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:41.050363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1085/2155] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:40 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:41.054813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c860c656-0ef7-40df-abd6-57053c59a06e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:41.116465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c860c656-0ef7-40df-abd6-57053c59a06e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:41.121664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc390e56-d360-4d12-b72b-f57e43e45d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:41.122268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc390e56-d360-4d12-b72b-f57e43e45d78 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:41.122550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc390e56-d360-4d12-b72b-f57e43e45d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:41.122826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc390e56-d360-4d12-b72b-f57e43e45d78 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:41.123533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc390e56-d360-4d12-b72b-f57e43e45d78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:41.125361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc390e56-d360-4d12-b72b-f57e43e45d78 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:41.249110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1071/2156] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:41 2022] POST /identity/v3/projects => generated 371 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:41.253198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f43638-f0e4-45b6-8919-216cb3676b00 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:41.264585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f43638-f0e4-45b6-8919-216cb3676b00 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:41.320363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca9e656-f4cb-4821-9749-ec3829487d7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:41.320997 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca9e656-f4cb-4821-9749-ec3829487d7a admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:41.321328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca9e656-f4cb-4821-9749-ec3829487d7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:41.321619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca9e656-f4cb-4821-9749-ec3829487d7a admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:41.322268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca9e656-f4cb-4821-9749-ec3829487d7a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:41.324173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca9e656-f4cb-4821-9749-ec3829487d7a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:41.334720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:41.479540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1086/2157] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:41 2022] POST /identity/v3/users => generated 292 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:41.481123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc390e56-d360-4d12-b72b-f57e43e45d78 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:41.491537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc390e56-d360-4d12-b72b-f57e43e45d78 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:41.496342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04bd0ff-4a9f-416a-8729-0b06c1d2ec54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:41.496946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04bd0ff-4a9f-416a-8729-0b06c1d2ec54 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:41.497231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04bd0ff-4a9f-416a-8729-0b06c1d2ec54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:41.497508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04bd0ff-4a9f-416a-8729-0b06c1d2ec54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:41.498250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04bd0ff-4a9f-416a-8729-0b06c1d2ec54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:41.499852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04bd0ff-4a9f-416a-8729-0b06c1d2ec54 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:41.511221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1072/2158] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:41.521534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca9e656-f4cb-4821-9749-ec3829487d7a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:41.527634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca9e656-f4cb-4821-9749-ec3829487d7a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:41.532284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9263ffaa-4911-4840-a79b-5b1215f03cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:41.533000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9263ffaa-4911-4840-a79b-5b1215f03cea admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:41.533284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9263ffaa-4911-4840-a79b-5b1215f03cea admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:41.533560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9263ffaa-4911-4840-a79b-5b1215f03cea admin admin] PATH_INFO: `/v3/projects/a89066d6e001406788ccecb5bf3a8cc7/users/5e2f248471fb4f598cf6f4492669b0c5/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:41.549141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9263ffaa-4911-4840-a79b-5b1215f03cea admin admin] RBAC: Authorizing `identity:create_grant(project_id=a89066d6e001406788ccecb5bf3a8cc7, user_id=5e2f248471fb4f598cf6f4492669b0c5, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:41.551865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9263ffaa-4911-4840-a79b-5b1215f03cea admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:41.653665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1087/2159] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:41 2022] PUT /identity/v3/projects/a89066d6e001406788ccecb5bf3a8cc7/users/5e2f248471fb4f598cf6f4492669b0c5/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:41.657510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04bd0ff-4a9f-416a-8729-0b06c1d2ec54 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:41.667912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04bd0ff-4a9f-416a-8729-0b06c1d2ec54 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:41.722695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a123de2b-22d4-4d30-8ec2-c9367369771d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:41.723294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123de2b-22d4-4d30-8ec2-c9367369771d admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:41.723574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123de2b-22d4-4d30-8ec2-c9367369771d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:41.723842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a123de2b-22d4-4d30-8ec2-c9367369771d admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:41.724530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a123de2b-22d4-4d30-8ec2-c9367369771d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:41.726140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a123de2b-22d4-4d30-8ec2-c9367369771d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:41.736400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1073/2160] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:41 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:41.740462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9263ffaa-4911-4840-a79b-5b1215f03cea admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:41.751212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9263ffaa-4911-4840-a79b-5b1215f03cea admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:41.755786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:41.756421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:41.756699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:41.756969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] PATH_INFO: `/v3/projects/a89066d6e001406788ccecb5bf3a8cc7/users/5e2f248471fb4f598cf6f4492669b0c5/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:41.758899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a89066d6e001406788ccecb5bf3a8cc7, user_id=5e2f248471fb4f598cf6f4492669b0c5, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:41.762219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:41.772237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1088/2161] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:41 2022] PUT /identity/v3/projects/a89066d6e001406788ccecb5bf3a8cc7/users/5e2f248471fb4f598cf6f4492669b0c5/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:41.782682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cdff2-0593-47da-8625-c6a90d0d9f59 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:41.782975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cdff2-0593-47da-8625-c6a90d0d9f59 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:41.783249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cdff2-0593-47da-8625-c6a90d0d9f59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:41.796523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:41.811390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f93cdff2-0593-47da-8625-c6a90d0d9f59 None None] MFA Rules not processed for user `5e2f248471fb4f598cf6f4492669b0c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:41.812605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f93cdff2-0593-47da-8625-c6a90d0d9f59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:41.880541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1074/2162] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:41 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:41.892026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:41.893759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:41.952907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6bdce2-b823-42ac-b76f-4a9068ccd3e6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:41.956547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b80b156-e633-45cb-bc10-d8886e8c9af3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:41.957147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80b156-e633-45cb-bc10-d8886e8c9af3 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:41.957430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80b156-e633-45cb-bc10-d8886e8c9af3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:41.957739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b80b156-e633-45cb-bc10-d8886e8c9af3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:41.966525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b80b156-e633-45cb-bc10-d8886e8c9af3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:41.968216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b80b156-e633-45cb-bc10-d8886e8c9af3 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:41.977315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1089/2163] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:41 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:42.875589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93cdff2-0593-47da-8625-c6a90d0d9f59 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:42.935891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93cdff2-0593-47da-8625-c6a90d0d9f59 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:42.940561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:42.941189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:42.941471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:42.941792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] PATH_INFO: `/v3/users/dec8c83eb6ea41edb22b57d77397e71b` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:42.942937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dec8c83eb6ea41edb22b57d77397e71b)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:42.944918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:43.112454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dec8c83eb6ea41edb22b57d77397e71b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:43.124567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dec8c83eb6ea41edb22b57d77397e71b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:43.125250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dec8c83eb6ea41edb22b57d77397e71b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:43.207336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dec8c83eb6ea41edb22b57d77397e71b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:43.215218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dec8c83eb6ea41edb22b57d77397e71b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:43.221425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1075/2164] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:42 2022] DELETE /identity/v3/users/dec8c83eb6ea41edb22b57d77397e71b => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:43.226020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e92699-3658-4dbb-be8a-96058c5142c7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:43.226420 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e92699-3658-4dbb-be8a-96058c5142c7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:43.228188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e92699-3658-4dbb-be8a-96058c5142c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:43.245243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-23e92699-3658-4dbb-be8a-96058c5142c7 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:43.246482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-23e92699-3658-4dbb-be8a-96058c5142c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:43.304602 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1090/2165] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:16:43.314745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:43.316151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:43.329917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:43.373801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d2b5d4-4c66-4727-bf4c-2778540c0405 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:43.377432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c12117-4ac8-40ae-abc7-ec72d307d8b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:43.378019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c12117-4ac8-40ae-abc7-ec72d307d8b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:43.378300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c12117-4ac8-40ae-abc7-ec72d307d8b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:43.378571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c12117-4ac8-40ae-abc7-ec72d307d8b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:43.387897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c12117-4ac8-40ae-abc7-ec72d307d8b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:43.389327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c12117-4ac8-40ae-abc7-ec72d307d8b8 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:43.399342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1076/2166] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:43.870080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e92699-3658-4dbb-be8a-96058c5142c7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:43.871512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-23e92699-3658-4dbb-be8a-96058c5142c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:43.884829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e92699-3658-4dbb-be8a-96058c5142c7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:43.891962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:43.891962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:43.891962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:43.891962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] PATH_INFO: `/v3/projects/50d92682bc5f4346ba53e5bb43ca210f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:43.893704 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50d92682bc5f4346ba53e5bb43ca210f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:43.895665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:44.047633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50d92682bc5f4346ba53e5bb43ca210f', 'request_id': 'req-c364956b-9a9e-4ea5-934e-829f76fa1fd2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.052221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:44.053491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50d92682bc5f4346ba53e5bb43ca210f', 'request_id': 'req-c364956b-9a9e-4ea5-934e-829f76fa1fd2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.127673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50d92682bc5f4346ba53e5bb43ca210f', 'request_id': 'req-c364956b-9a9e-4ea5-934e-829f76fa1fd2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.139426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] The token cache is being invalidate because project 50d92682bc5f4346ba53e5bb43ca210f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:44.139712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.140340 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.141906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1091/2167] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:43 2022] DELETE /identity/v3/projects/50d92682bc5f4346ba53e5bb43ca210f => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:44.155732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c12117-4ac8-40ae-abc7-ec72d307d8b8 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:44.165579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c12117-4ac8-40ae-abc7-ec72d307d8b8 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:44.222132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:44.222762 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:44.223045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:44.223324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] PATH_INFO: `/v3/users/5e2f248471fb4f598cf6f4492669b0c5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:44.224455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e2f248471fb4f598cf6f4492669b0c5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:44.226435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:44.490111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e2f248471fb4f598cf6f4492669b0c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.501854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e2f248471fb4f598cf6f4492669b0c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.502543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e2f248471fb4f598cf6f4492669b0c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.716043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e2f248471fb4f598cf6f4492669b0c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.724148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-51fc27bf-27c8-4e61-9c91-0460b7a1a71c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e2f248471fb4f598cf6f4492669b0c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:44.730764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1077/2168] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:44 2022] DELETE /identity/v3/users/5e2f248471fb4f598cf6f4492669b0c5 => generated 0 bytes in 575 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:45.163887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:45.169054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:45.180083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c364956b-9a9e-4ea5-934e-829f76fa1fd2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:45.243391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:45.244023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:45.244858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:45.245133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] PATH_INFO: `/v3/projects/a89066d6e001406788ccecb5bf3a8cc7` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:45.247353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a89066d6e001406788ccecb5bf3a8cc7)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:45.249218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:45.403477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a89066d6e001406788ccecb5bf3a8cc7', 'request_id': 'req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:45.408883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a89066d6e001406788ccecb5bf3a8cc7', 'request_id': 'req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:45.527715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a89066d6e001406788ccecb5bf3a8cc7', 'request_id': 'req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:45.539600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] The token cache is being invalidate because project a89066d6e001406788ccecb5bf3a8cc7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:45.539884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:45.540507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-3b42f5ea-d0b6-4ddc-915e-09030fe372c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:45.542607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1092/2169] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:45 2022] DELETE /identity/v3/projects/a89066d6e001406788ccecb5bf3a8cc7 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:45.547623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9804b20-da03-4609-8200-bce1b4199c3d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:45.548064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9804b20-da03-4609-8200-bce1b4199c3d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:45.548180 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9804b20-da03-4609-8200-bce1b4199c3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:45.567789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-b9804b20-da03-4609-8200-bce1b4199c3d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:45.568971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b9804b20-da03-4609-8200-bce1b4199c3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:45.598174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:45.627950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1078/2170] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:45.635963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25e7905-83e7-4c87-9631-b2df490f2d68 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:45.636251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25e7905-83e7-4c87-9631-b2df490f2d68 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:45.640548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25e7905-83e7-4c87-9631-b2df490f2d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:45.656833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b25e7905-83e7-4c87-9631-b2df490f2d68 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:45.658101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b25e7905-83e7-4c87-9631-b2df490f2d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:45.666875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1093/2171] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:45.671177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9804b20-da03-4609-8200-bce1b4199c3d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:45.680884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9804b20-da03-4609-8200-bce1b4199c3d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:45.685700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:45.686296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:45.686578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:45.686852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:45.687499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:16:45.687805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:45.689489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:45.700801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1079/2172] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:16:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:16:45.706660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25e7905-83e7-4c87-9631-b2df490f2d68 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:45.716591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25e7905-83e7-4c87-9631-b2df490f2d68 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:45.721640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e8323a-2a7e-456c-b10c-877f8e9551f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:45.722239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8323a-2a7e-456c-b10c-877f8e9551f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:45.722513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8323a-2a7e-456c-b10c-877f8e9551f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:45.722780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8323a-2a7e-456c-b10c-877f8e9551f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:45.723400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e8323a-2a7e-456c-b10c-877f8e9551f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:45.725282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e8323a-2a7e-456c-b10c-877f8e9551f2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:45.814538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1094/2173] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:45 2022] POST /identity/v3/projects => generated 369 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:16:45.818808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:45.828737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1cba3b-261d-4934-93e0-b1d244615186 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:45.886870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91484ced-b4d7-4058-bdef-34a9cd6562dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:45.887430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91484ced-b4d7-4058-bdef-34a9cd6562dc admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:45.887711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91484ced-b4d7-4058-bdef-34a9cd6562dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:45.887983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91484ced-b4d7-4058-bdef-34a9cd6562dc admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:45.888611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91484ced-b4d7-4058-bdef-34a9cd6562dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:45.890673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91484ced-b4d7-4058-bdef-34a9cd6562dc admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:45.990760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1080/2174] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:45 2022] POST /identity/v3/users => generated 285 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:45.994900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e8323a-2a7e-456c-b10c-877f8e9551f2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.005278 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e8323a-2a7e-456c-b10c-877f8e9551f2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.009912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e53216b-5f00-49df-984c-216340207cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.010499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e53216b-5f00-49df-984c-216340207cdd admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.010827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e53216b-5f00-49df-984c-216340207cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.011098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e53216b-5f00-49df-984c-216340207cdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.011738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e53216b-5f00-49df-984c-216340207cdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.013321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e53216b-5f00-49df-984c-216340207cdd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.023329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1095/2175] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:46.027911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91484ced-b4d7-4058-bdef-34a9cd6562dc admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.038038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91484ced-b4d7-4058-bdef-34a9cd6562dc admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.043369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b597e0-1ccb-4d38-8ff0-1e2961253ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.043960 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b597e0-1ccb-4d38-8ff0-1e2961253ea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.044241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b597e0-1ccb-4d38-8ff0-1e2961253ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.044515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b597e0-1ccb-4d38-8ff0-1e2961253ea5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.045203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b597e0-1ccb-4d38-8ff0-1e2961253ea5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.046836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b597e0-1ccb-4d38-8ff0-1e2961253ea5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.056580 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1081/2176] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:46.061392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e53216b-5f00-49df-984c-216340207cdd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.071803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e53216b-5f00-49df-984c-216340207cdd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.076982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786e63ae-e5c9-4b29-bd47-65d74f5c8203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.077655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786e63ae-e5c9-4b29-bd47-65d74f5c8203 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.077939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786e63ae-e5c9-4b29-bd47-65d74f5c8203 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.078207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786e63ae-e5c9-4b29-bd47-65d74f5c8203 admin admin] PATH_INFO: `/v3/projects/1290f141e325435cb6957123e8825bf8/users/a8f40581f0744c459e5fcf63990d3060/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.092860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786e63ae-e5c9-4b29-bd47-65d74f5c8203 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1290f141e325435cb6957123e8825bf8, user_id=a8f40581f0744c459e5fcf63990d3060, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.095523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786e63ae-e5c9-4b29-bd47-65d74f5c8203 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.179353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1096/2177] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:46 2022] PUT /identity/v3/projects/1290f141e325435cb6957123e8825bf8/users/a8f40581f0744c459e5fcf63990d3060/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:16:46.190404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff31f32-1482-4aa4-9a00-75396af35b17 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.190693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff31f32-1482-4aa4-9a00-75396af35b17 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.190971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff31f32-1482-4aa4-9a00-75396af35b17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.220052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-dff31f32-1482-4aa4-9a00-75396af35b17 None None] MFA Rules not processed for user `a8f40581f0744c459e5fcf63990d3060`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:46.221272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-dff31f32-1482-4aa4-9a00-75396af35b17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:46.294124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1082/2178] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:46 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:46.298684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786e63ae-e5c9-4b29-bd47-65d74f5c8203 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.312222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:46.361240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786e63ae-e5c9-4b29-bd47-65d74f5c8203 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.366088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2b1dc8-4af0-495f-9e02-8b527736cb59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.366689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2b1dc8-4af0-495f-9e02-8b527736cb59 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.366967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2b1dc8-4af0-495f-9e02-8b527736cb59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.367239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2b1dc8-4af0-495f-9e02-8b527736cb59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.367855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2b1dc8-4af0-495f-9e02-8b527736cb59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.370082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2b1dc8-4af0-495f-9e02-8b527736cb59 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.477533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1097/2179] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:46 2022] POST /identity/v3/projects => generated 369 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:46.481570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff31f32-1482-4aa4-9a00-75396af35b17 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.492247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff31f32-1482-4aa4-9a00-75396af35b17 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.518240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:46.548010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833ec84-78ca-4d3a-8ac6-590014a33dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.548704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833ec84-78ca-4d3a-8ac6-590014a33dbe admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.548913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833ec84-78ca-4d3a-8ac6-590014a33dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.549188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8833ec84-78ca-4d3a-8ac6-590014a33dbe admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.549858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8833ec84-78ca-4d3a-8ac6-590014a33dbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.551772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8833ec84-78ca-4d3a-8ac6-590014a33dbe admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.659364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1083/2180] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:46 2022] POST /identity/v3/users => generated 291 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:46.663292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2b1dc8-4af0-495f-9e02-8b527736cb59 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.674029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2b1dc8-4af0-495f-9e02-8b527736cb59 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.678664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90812d6-9da7-4ef6-b530-a779a56f7cca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.679253 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90812d6-9da7-4ef6-b530-a779a56f7cca admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.679531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90812d6-9da7-4ef6-b530-a779a56f7cca admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.679803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90812d6-9da7-4ef6-b530-a779a56f7cca admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.680469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90812d6-9da7-4ef6-b530-a779a56f7cca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.682745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90812d6-9da7-4ef6-b530-a779a56f7cca admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.693155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1098/2181] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:46.697287 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833ec84-78ca-4d3a-8ac6-590014a33dbe admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.707403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8833ec84-78ca-4d3a-8ac6-590014a33dbe admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.712025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8299bcf0-9d0f-427f-9e79-5dc868a0e65c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.712678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8299bcf0-9d0f-427f-9e79-5dc868a0e65c admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.713471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8299bcf0-9d0f-427f-9e79-5dc868a0e65c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.715811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8299bcf0-9d0f-427f-9e79-5dc868a0e65c admin admin] PATH_INFO: `/v3/projects/74b439c04f5d451891aa73fc69e4dc87/users/04e645dec08f44dfbce917725ad5ce41/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.730333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8299bcf0-9d0f-427f-9e79-5dc868a0e65c admin admin] RBAC: Authorizing `identity:create_grant(project_id=74b439c04f5d451891aa73fc69e4dc87, user_id=04e645dec08f44dfbce917725ad5ce41, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.733079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8299bcf0-9d0f-427f-9e79-5dc868a0e65c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.854708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1084/2182] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:46 2022] PUT /identity/v3/projects/74b439c04f5d451891aa73fc69e4dc87/users/04e645dec08f44dfbce917725ad5ce41/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:46.858682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90812d6-9da7-4ef6-b530-a779a56f7cca admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.868702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90812d6-9da7-4ef6-b530-a779a56f7cca admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.928618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b105cf3-5019-4c5b-b2e3-e56118f989fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.929384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b105cf3-5019-4c5b-b2e3-e56118f989fc admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.929723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b105cf3-5019-4c5b-b2e3-e56118f989fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.930002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b105cf3-5019-4c5b-b2e3-e56118f989fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.930669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b105cf3-5019-4c5b-b2e3-e56118f989fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.932332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b105cf3-5019-4c5b-b2e3-e56118f989fc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.945628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1099/2183] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:46 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:46.946543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8299bcf0-9d0f-427f-9e79-5dc868a0e65c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:46.972943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8299bcf0-9d0f-427f-9e79-5dc868a0e65c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:46.978201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:46.980222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:46.980222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:46.980222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] PATH_INFO: `/v3/projects/74b439c04f5d451891aa73fc69e4dc87/users/04e645dec08f44dfbce917725ad5ce41/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:46.982341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74b439c04f5d451891aa73fc69e4dc87, user_id=04e645dec08f44dfbce917725ad5ce41, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:46.985100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:46.999125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1085/2184] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:46 2022] PUT /identity/v3/projects/74b439c04f5d451891aa73fc69e4dc87/users/04e645dec08f44dfbce917725ad5ce41/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:47.009569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f2d227-a763-4897-ae89-3083d89f1578 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:47.009959 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f2d227-a763-4897-ae89-3083d89f1578 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:47.010238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f2d227-a763-4897-ae89-3083d89f1578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:47.040208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-73f2d227-a763-4897-ae89-3083d89f1578 None None] MFA Rules not processed for user `04e645dec08f44dfbce917725ad5ce41`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:47.042507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-73f2d227-a763-4897-ae89-3083d89f1578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:47.123290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1100/2185] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:47 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:47.134478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:47.136220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:47.199566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc1d5c0-e4d5-494c-8ad5-d23b48222679 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:47.203540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63996787-c618-417d-be5d-14d5d14f1778 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:47.204187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63996787-c618-417d-be5d-14d5d14f1778 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:47.204469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63996787-c618-417d-be5d-14d5d14f1778 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:47.204745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63996787-c618-417d-be5d-14d5d14f1778 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:47.214165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63996787-c618-417d-be5d-14d5d14f1778 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:47.215638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63996787-c618-417d-be5d-14d5d14f1778 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:47.227250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1086/2186] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:47 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:47.819262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f2d227-a763-4897-ae89-3083d89f1578 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:47.821165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-73f2d227-a763-4897-ae89-3083d89f1578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:47.832990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:47.888546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f2d227-a763-4897-ae89-3083d89f1578 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:47.893073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:47.893736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:47.894012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:47.894286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] PATH_INFO: `/v3/users/deae76d7994f40a69b4041fdfdf499d8` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:47.895388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] RBAC: Authorizing `identity:delete_user(user_id=deae76d7994f40a69b4041fdfdf499d8)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:47.897332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:48.093351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'deae76d7994f40a69b4041fdfdf499d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:48.094088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'deae76d7994f40a69b4041fdfdf499d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:48.107816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'deae76d7994f40a69b4041fdfdf499d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:48.184068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'deae76d7994f40a69b4041fdfdf499d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:48.191924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'deae76d7994f40a69b4041fdfdf499d8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:48.199110 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1101/2187] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:47 2022] DELETE /identity/v3/users/deae76d7994f40a69b4041fdfdf499d8 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:48.202865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136282a7-7440-4e08-9cd0-3689e34fd57d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:48.203343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136282a7-7440-4e08-9cd0-3689e34fd57d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:48.203623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136282a7-7440-4e08-9cd0-3689e34fd57d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:48.225985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-136282a7-7440-4e08-9cd0-3689e34fd57d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:48.226522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-136282a7-7440-4e08-9cd0-3689e34fd57d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:48.286821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1087/2188] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:16:48.297097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:48.298601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:48.359261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9baaed-1a18-48c6-9a0c-6adb9451957b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:48.363209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:48.363858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:48.364175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:48.364454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:48.373884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:48.375471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:48.385192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1102/2189] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:48.861481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136282a7-7440-4e08-9cd0-3689e34fd57d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:48.863174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-136282a7-7440-4e08-9cd0-3689e34fd57d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:48.881709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136282a7-7440-4e08-9cd0-3689e34fd57d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:48.881709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:48.881709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:48.881709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:48.882121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] PATH_INFO: `/v3/projects/a134dff9974748abbd1d559a7f822708` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:48.883007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a134dff9974748abbd1d559a7f822708)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:48.885510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:48.908890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:49.030821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a134dff9974748abbd1d559a7f822708', 'request_id': 'req-cb7151c9-0c9d-4587-9d90-83eeb64b4682'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:49.105705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a134dff9974748abbd1d559a7f822708', 'request_id': 'req-cb7151c9-0c9d-4587-9d90-83eeb64b4682'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:49.112722 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a134dff9974748abbd1d559a7f822708', 'request_id': 'req-cb7151c9-0c9d-4587-9d90-83eeb64b4682'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:49.124928 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] The token cache is being invalidate because project a134dff9974748abbd1d559a7f822708 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:49.125217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:49.125955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-cb7151c9-0c9d-4587-9d90-83eeb64b4682 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:49.127584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1088/2190] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:48 2022] DELETE /identity/v3/projects/a134dff9974748abbd1d559a7f822708 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:50.502581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:50.504010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:50.516680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee2058-7922-4907-a39e-ad88fe8910a5 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:50.543337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:50.573668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:50.574291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:50.574568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:50.574832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] PATH_INFO: `/v3/users/a8f40581f0744c459e5fcf63990d3060` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:50.575921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8f40581f0744c459e5fcf63990d3060)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:50.577906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:50.786447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8f40581f0744c459e5fcf63990d3060'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:50.787033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8f40581f0744c459e5fcf63990d3060'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:50.799355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8f40581f0744c459e5fcf63990d3060'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:50.870514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8f40581f0744c459e5fcf63990d3060'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:50.877528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8f40581f0744c459e5fcf63990d3060'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:50.883377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1103/2191] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:50 2022] DELETE /identity/v3/users/a8f40581f0744c459e5fcf63990d3060 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:50.888370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f7991-f101-4234-872f-c3529d94c027 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:50.888838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f7991-f101-4234-872f-c3529d94c027 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:50.888931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f7991-f101-4234-872f-c3529d94c027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:50.908992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e22f7991-f101-4234-872f-c3529d94c027 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:50.910296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e22f7991-f101-4234-872f-c3529d94c027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:50.970615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1089/2192] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:50.980430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:50.981961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:51.039690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc191c2f-7577-4319-afd9-214f412cbbee admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:51.043199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd6881-a852-4045-9768-50a8364f5e95 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:51.043806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd6881-a852-4045-9768-50a8364f5e95 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:51.044083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd6881-a852-4045-9768-50a8364f5e95 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:51.044353 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd6881-a852-4045-9768-50a8364f5e95 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:51.053489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcd6881-a852-4045-9768-50a8364f5e95 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:51.054948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcd6881-a852-4045-9768-50a8364f5e95 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:51.064360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1104/2193] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:16:51.540357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22f7991-f101-4234-872f-c3529d94c027 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:51.542581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e22f7991-f101-4234-872f-c3529d94c027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:51.554411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22f7991-f101-4234-872f-c3529d94c027 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:51.559515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:51.560314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:51.564733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:51.564733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] PATH_INFO: `/v3/projects/1290f141e325435cb6957123e8825bf8` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:51.564733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] RBAC: Authorizing `identity:delete_project(project_id=1290f141e325435cb6957123e8825bf8)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:51.564733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:51.688247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1290f141e325435cb6957123e8825bf8', 'request_id': 'req-ce864dd7-5819-4464-886f-f2b5f9707ddc'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:51.778972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1290f141e325435cb6957123e8825bf8', 'request_id': 'req-ce864dd7-5819-4464-886f-f2b5f9707ddc'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:51.784857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1290f141e325435cb6957123e8825bf8', 'request_id': 'req-ce864dd7-5819-4464-886f-f2b5f9707ddc'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:51.796192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] The token cache is being invalidate because project 1290f141e325435cb6957123e8825bf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:51.796481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:51.797111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:51.798613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1090/2194] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:51 2022] DELETE /identity/v3/projects/1290f141e325435cb6957123e8825bf8 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:51.802644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd6881-a852-4045-9768-50a8364f5e95 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:51.812378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd6881-a852-4045-9768-50a8364f5e95 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:51.859327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:51.868566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:51.869189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:51.869464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:51.869767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] PATH_INFO: `/v3/users/04e645dec08f44dfbce917725ad5ce41` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:51.870893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04e645dec08f44dfbce917725ad5ce41)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:51.872787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:52.129251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04e645dec08f44dfbce917725ad5ce41'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.129919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04e645dec08f44dfbce917725ad5ce41'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.142354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04e645dec08f44dfbce917725ad5ce41'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.230724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04e645dec08f44dfbce917725ad5ce41'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.238425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-914abc07-3729-4b77-92a1-77f82c441448 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04e645dec08f44dfbce917725ad5ce41'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.244325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1105/2195] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:16:51 2022] DELETE /identity/v3/users/04e645dec08f44dfbce917725ad5ce41 => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:52.644155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:52.645648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:52.659812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce864dd7-5819-4464-886f-f2b5f9707ddc admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:52.694506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:52.724135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:52.724767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:52.725048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:52.725320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] PATH_INFO: `/v3/projects/74b439c04f5d451891aa73fc69e4dc87` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:52.726513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74b439c04f5d451891aa73fc69e4dc87)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:52.728559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:52.864999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74b439c04f5d451891aa73fc69e4dc87', 'request_id': 'req-5290c28f-b432-4747-bb93-b70198007623'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.961565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74b439c04f5d451891aa73fc69e4dc87', 'request_id': 'req-5290c28f-b432-4747-bb93-b70198007623'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.967671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74b439c04f5d451891aa73fc69e4dc87', 'request_id': 'req-5290c28f-b432-4747-bb93-b70198007623'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.978660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] The token cache is being invalidate because project 74b439c04f5d451891aa73fc69e4dc87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:16:52.978945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.979564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5290c28f-b432-4747-bb93-b70198007623 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:16:52.981168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1091/2196] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:16:52 2022] DELETE /identity/v3/projects/74b439c04f5d451891aa73fc69e4dc87 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:52.986100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af090961-9f99-4078-a40b-059c16d93739 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:52.986589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af090961-9f99-4078-a40b-059c16d93739 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:52.986685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af090961-9f99-4078-a40b-059c16d93739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.006290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-af090961-9f99-4078-a40b-059c16d93739 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:53.007472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-af090961-9f99-4078-a40b-059c16d93739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:53.065342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1106/2197] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:53.074381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408f4b-20ee-48fc-bddb-1c5b72f507bd None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.074672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408f4b-20ee-48fc-bddb-1c5b72f507bd None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.074949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68408f4b-20ee-48fc-bddb-1c5b72f507bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.095108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-68408f4b-20ee-48fc-bddb-1c5b72f507bd None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:53.096325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-68408f4b-20ee-48fc-bddb-1c5b72f507bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:53.105099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1092/2198] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:53.109460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af090961-9f99-4078-a40b-059c16d93739 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:53.119988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af090961-9f99-4078-a40b-059c16d93739 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:53.124467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:53.125063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.125341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.125634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.126279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:16:53.126581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:53.128195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:53.140011 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1107/2199] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:16:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:16:53.144216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68408f4b-20ee-48fc-bddb-1c5b72f507bd None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:53.153926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68408f4b-20ee-48fc-bddb-1c5b72f507bd None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:53.158542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cb3b49-de4b-4f14-b658-e5b6bf7597ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:53.159129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb3b49-de4b-4f14-b658-e5b6bf7597ee admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.159409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb3b49-de4b-4f14-b658-e5b6bf7597ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.159679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb3b49-de4b-4f14-b658-e5b6bf7597ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.160295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cb3b49-de4b-4f14-b658-e5b6bf7597ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:53.162234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cb3b49-de4b-4f14-b658-e5b6bf7597ee admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:53.254795 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1093/2200] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:53 2022] POST /identity/v3/projects => generated 369 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:53.259040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:53.269258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2f5b24-f1cc-4482-8d5a-b9697cbc57c7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:53.320590 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:53.326614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1918ec1f-dd86-4d01-bc89-e63986b9ba69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:53.327217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1918ec1f-dd86-4d01-bc89-e63986b9ba69 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.327492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1918ec1f-dd86-4d01-bc89-e63986b9ba69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.327761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1918ec1f-dd86-4d01-bc89-e63986b9ba69 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.328387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1918ec1f-dd86-4d01-bc89-e63986b9ba69 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:53.330377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1918ec1f-dd86-4d01-bc89-e63986b9ba69 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:53.442568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1108/2201] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:53 2022] POST /identity/v3/users => generated 285 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:53.446954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cb3b49-de4b-4f14-b658-e5b6bf7597ee admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:53.457465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cb3b49-de4b-4f14-b658-e5b6bf7597ee admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:53.462131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48be30a2-47e8-4a65-a6a4-97fdec68d8b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:53.463274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48be30a2-47e8-4a65-a6a4-97fdec68d8b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.463555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48be30a2-47e8-4a65-a6a4-97fdec68d8b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.463827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48be30a2-47e8-4a65-a6a4-97fdec68d8b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.464469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48be30a2-47e8-4a65-a6a4-97fdec68d8b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:53.466089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48be30a2-47e8-4a65-a6a4-97fdec68d8b4 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:53.479369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1094/2202] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:53.481864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1918ec1f-dd86-4d01-bc89-e63986b9ba69 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:53.491726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1918ec1f-dd86-4d01-bc89-e63986b9ba69 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:53.496326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2511d5-8a0d-45e7-901c-724fe7475735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:53.496924 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2511d5-8a0d-45e7-901c-724fe7475735 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.497195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2511d5-8a0d-45e7-901c-724fe7475735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.497462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2511d5-8a0d-45e7-901c-724fe7475735 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.498135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2511d5-8a0d-45e7-901c-724fe7475735 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:53.499703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2511d5-8a0d-45e7-901c-724fe7475735 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:53.510415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1109/2203] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:53.514711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48be30a2-47e8-4a65-a6a4-97fdec68d8b4 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:53.524898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48be30a2-47e8-4a65-a6a4-97fdec68d8b4 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:53.530245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd9af01-21b6-435c-be5c-5427a2434c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:53.530932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd9af01-21b6-435c-be5c-5427a2434c7d admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.531208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd9af01-21b6-435c-be5c-5427a2434c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.531506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd9af01-21b6-435c-be5c-5427a2434c7d admin admin] PATH_INFO: `/v3/projects/0cd491659d62499789879d73a1c09dc3/users/44f557a478cf4695a1c5155fa513d97b/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.546968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd9af01-21b6-435c-be5c-5427a2434c7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0cd491659d62499789879d73a1c09dc3, user_id=44f557a478cf4695a1c5155fa513d97b, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:53.549638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd9af01-21b6-435c-be5c-5427a2434c7d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:53.665401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1095/2204] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:53 2022] PUT /identity/v3/projects/0cd491659d62499789879d73a1c09dc3/users/44f557a478cf4695a1c5155fa513d97b/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:53.676566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42677289-ca3e-4c3f-96b9-093f3a14843e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.676858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42677289-ca3e-4c3f-96b9-093f3a14843e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.677135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42677289-ca3e-4c3f-96b9-093f3a14843e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.705379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-42677289-ca3e-4c3f-96b9-093f3a14843e None None] MFA Rules not processed for user `44f557a478cf4695a1c5155fa513d97b`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:53.706762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-42677289-ca3e-4c3f-96b9-093f3a14843e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:53.770667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1110/2205] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:53 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:53.775820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd9af01-21b6-435c-be5c-5427a2434c7d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:53.836660 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:53.841495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd9af01-21b6-435c-be5c-5427a2434c7d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:53.846411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89760c5c-ae22-49d2-b50e-616e04626a09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:53.850451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89760c5c-ae22-49d2-b50e-616e04626a09 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:53.850451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89760c5c-ae22-49d2-b50e-616e04626a09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:53.850451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89760c5c-ae22-49d2-b50e-616e04626a09 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:53.850451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89760c5c-ae22-49d2-b50e-616e04626a09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:53.850451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89760c5c-ae22-49d2-b50e-616e04626a09 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:53.986409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1096/2206] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:16:53 2022] POST /identity/v3/projects => generated 367 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:53.990825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42677289-ca3e-4c3f-96b9-093f3a14843e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:54.002432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42677289-ca3e-4c3f-96b9-093f3a14843e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:54.061781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a6af2-5336-4fc5-897b-0642621a60d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:54.062387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a6af2-5336-4fc5-897b-0642621a60d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.062663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a6af2-5336-4fc5-897b-0642621a60d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.062934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a6af2-5336-4fc5-897b-0642621a60d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.063561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424a6af2-5336-4fc5-897b-0642621a60d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:54.065471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424a6af2-5336-4fc5-897b-0642621a60d2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:54.162358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1111/2207] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:16:53 2022] POST /identity/v3/users => generated 290 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:16:54.171890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89760c5c-ae22-49d2-b50e-616e04626a09 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:54.183846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89760c5c-ae22-49d2-b50e-616e04626a09 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:54.188915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169c3c7f-e033-4af9-9e0f-538e23359932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:54.189666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169c3c7f-e033-4af9-9e0f-538e23359932 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.190126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169c3c7f-e033-4af9-9e0f-538e23359932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.190393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169c3c7f-e033-4af9-9e0f-538e23359932 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.191032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169c3c7f-e033-4af9-9e0f-538e23359932 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:54.192697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-169c3c7f-e033-4af9-9e0f-538e23359932 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:54.203563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1097/2208] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:54 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:54.207772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a6af2-5336-4fc5-897b-0642621a60d2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:54.217895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a6af2-5336-4fc5-897b-0642621a60d2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:54.222519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3669f9-173c-43a6-beb5-8b47b42a7300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:54.223209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3669f9-173c-43a6-beb5-8b47b42a7300 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.223484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3669f9-173c-43a6-beb5-8b47b42a7300 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.224183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3669f9-173c-43a6-beb5-8b47b42a7300 admin admin] PATH_INFO: `/v3/projects/523140ae5d6e48858c9bc29e80fc5b04/users/276e6f4c4e62466db82a2722be259c4e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.239386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3669f9-173c-43a6-beb5-8b47b42a7300 admin admin] RBAC: Authorizing `identity:create_grant(project_id=523140ae5d6e48858c9bc29e80fc5b04, user_id=276e6f4c4e62466db82a2722be259c4e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:54.242197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3669f9-173c-43a6-beb5-8b47b42a7300 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:54.344948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1112/2209] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:54 2022] PUT /identity/v3/projects/523140ae5d6e48858c9bc29e80fc5b04/users/276e6f4c4e62466db82a2722be259c4e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:54.349084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169c3c7f-e033-4af9-9e0f-538e23359932 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:54.359121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-169c3c7f-e033-4af9-9e0f-538e23359932 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:54.415201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:54.415822 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.416170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.416443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.417080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:54.418709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:54.429221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1098/2210] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:16:54 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:16:54.433837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3669f9-173c-43a6-beb5-8b47b42a7300 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:54.444077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3669f9-173c-43a6-beb5-8b47b42a7300 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:54.448455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:54.449722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4f626f-d28a-41ce-9424-80c7eb7d7568 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:54.455395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:54.455508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f626f-d28a-41ce-9424-80c7eb7d7568 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.455508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f626f-d28a-41ce-9424-80c7eb7d7568 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.455508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4f626f-d28a-41ce-9424-80c7eb7d7568 admin admin] PATH_INFO: `/v3/projects/523140ae5d6e48858c9bc29e80fc5b04/users/276e6f4c4e62466db82a2722be259c4e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.455508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4f626f-d28a-41ce-9424-80c7eb7d7568 admin admin] RBAC: Authorizing `identity:create_grant(project_id=523140ae5d6e48858c9bc29e80fc5b04, user_id=276e6f4c4e62466db82a2722be259c4e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:54.455508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4f626f-d28a-41ce-9424-80c7eb7d7568 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:54.465809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1113/2211] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:16:54 2022] PUT /identity/v3/projects/523140ae5d6e48858c9bc29e80fc5b04/users/276e6f4c4e62466db82a2722be259c4e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:16:54.476849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de3b905-7287-4048-ac50-6da255ef51bc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.477135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de3b905-7287-4048-ac50-6da255ef51bc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.478171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de3b905-7287-4048-ac50-6da255ef51bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.505430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1de3b905-7287-4048-ac50-6da255ef51bc None None] MFA Rules not processed for user `276e6f4c4e62466db82a2722be259c4e`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:54.506598 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1de3b905-7287-4048-ac50-6da255ef51bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:54.512642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:54.522522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff692b8-7af1-42f7-b263-bc7eba973419 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:54.577053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1114/2212] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:16:54 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:16:54.582152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:54.582895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.583181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.583454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.585203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:54.588513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de3b905-7287-4048-ac50-6da255ef51bc None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:54.589843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1de3b905-7287-4048-ac50-6da255ef51bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:54.597197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:54.598880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:54.605006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:16:54.609879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1099/2213] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:16:54.646021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de3b905-7287-4048-ac50-6da255ef51bc None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:54.649834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70855b1-106f-42d0-b4f0-0da09b0641b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:54.651047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70855b1-106f-42d0-b4f0-0da09b0641b9 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.651320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70855b1-106f-42d0-b4f0-0da09b0641b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.651582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70855b1-106f-42d0-b4f0-0da09b0641b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.660530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70855b1-106f-42d0-b4f0-0da09b0641b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:54.661971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70855b1-106f-42d0-b4f0-0da09b0641b9 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:54.672160 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1115/2214] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:54 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:16:54.774705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:54.776126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:54.787195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8494abd9-48fe-4ced-a665-bce103a8307a service placement] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:54.791416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:54.792000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:54.792273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:54.792538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:54.847896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:54.849315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:54.859079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1100/2215] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:16:54 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:16:56.349339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d7a19-6522-4127-beeb-284a5e8f69b0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:56.349814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d7a19-6522-4127-beeb-284a5e8f69b0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:56.349946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d7a19-6522-4127-beeb-284a5e8f69b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:56.368920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-9c6d7a19-6522-4127-beeb-284a5e8f69b0 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:56.370143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9c6d7a19-6522-4127-beeb-284a5e8f69b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:56.427311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1116/2216] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:16:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:56.439585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:56.441122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:56.511731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae6e727-b828-4c68-8267-c0f8c9017e2e service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:56.516843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7eeaa2-f378-4677-91da-225e50ae6f90 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:56.517478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7eeaa2-f378-4677-91da-225e50ae6f90 service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:56.517790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7eeaa2-f378-4677-91da-225e50ae6f90 service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:56.518066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7eeaa2-f378-4677-91da-225e50ae6f90 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:56.526230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7eeaa2-f378-4677-91da-225e50ae6f90 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:56.527853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7eeaa2-f378-4677-91da-225e50ae6f90 service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:56.536298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1101/2217] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:16:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:16:57.529561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f01b39-6015-4c0c-923e-2ac450b4d7f8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:57.530075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f01b39-6015-4c0c-923e-2ac450b4d7f8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:57.530174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f01b39-6015-4c0c-923e-2ac450b4d7f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:57.549503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-64f01b39-6015-4c0c-923e-2ac450b4d7f8 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:57.550763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-64f01b39-6015-4c0c-923e-2ac450b4d7f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:57.558525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1117/2218] 10.222.0.16 () {58 vars in 967 bytes} [Thu Aug 25 05:16:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:57.570414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7eeaa2-f378-4677-91da-225e50ae6f90 service swift] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:57.580791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7eeaa2-f378-4677-91da-225e50ae6f90 service swift] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:57.585907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d66a1-c9c4-450e-9076-a29245e55694 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:57.586494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d66a1-c9c4-450e-9076-a29245e55694 service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:57.588998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d66a1-c9c4-450e-9076-a29245e55694 service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:57.588998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35d66a1-c9c4-450e-9076-a29245e55694 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:57.595331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35d66a1-c9c4-450e-9076-a29245e55694 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:57.596890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35d66a1-c9c4-450e-9076-a29245e55694 service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:57.605234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1102/2219] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:16:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:16:58.265628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce96ab-e958-4a7a-bd42-80323534726a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:58.265921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce96ab-e958-4a7a-bd42-80323534726a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:58.266222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce96ab-e958-4a7a-bd42-80323534726a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:58.286216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4dce96ab-e958-4a7a-bd42-80323534726a None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:58.287403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4dce96ab-e958-4a7a-bd42-80323534726a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:58.295025 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1118/2220] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:16:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:58.307307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d66a1-c9c4-450e-9076-a29245e55694 service swift] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:58.317743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35d66a1-c9c4-450e-9076-a29245e55694 service swift] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:58.322819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea22a733-54ee-4d88-92f6-fb66ebca87ea service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:58.323402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22a733-54ee-4d88-92f6-fb66ebca87ea service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:58.323678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22a733-54ee-4d88-92f6-fb66ebca87ea service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:58.323946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22a733-54ee-4d88-92f6-fb66ebca87ea service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:58.331735 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea22a733-54ee-4d88-92f6-fb66ebca87ea service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:58.333291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea22a733-54ee-4d88-92f6-fb66ebca87ea service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:58.341197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1103/2221] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:16:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:16:58.927116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ac346-8497-47d8-b781-90711f19c874 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:58.927582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ac346-8497-47d8-b781-90711f19c874 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:58.927582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ac346-8497-47d8-b781-90711f19c874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:58.946312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-650ac346-8497-47d8-b781-90711f19c874 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:16:58.947509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-650ac346-8497-47d8-b781-90711f19c874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:16:58.954756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1119/2222] 10.222.0.16 () {58 vars in 967 bytes} [Thu Aug 25 05:16:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:16:58.968244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea22a733-54ee-4d88-92f6-fb66ebca87ea service swift] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:16:58.978556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea22a733-54ee-4d88-92f6-fb66ebca87ea service swift] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:16:58.983970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c25a0bd-618f-40d2-a437-3eec67364adf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:16:58.988350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25a0bd-618f-40d2-a437-3eec67364adf service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:16:58.988350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25a0bd-618f-40d2-a437-3eec67364adf service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:16:58.988350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c25a0bd-618f-40d2-a437-3eec67364adf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:16:58.992872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c25a0bd-618f-40d2-a437-3eec67364adf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:16:58.994480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c25a0bd-618f-40d2-a437-3eec67364adf service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:16:59.002532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1104/2223] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:16:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:17:00.027891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650ac346-8497-47d8-b781-90711f19c874 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:00.061028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:00.089631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650ac346-8497-47d8-b781-90711f19c874 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:00.094393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:00.095013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:00.095291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:00.095563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] PATH_INFO: `/v3/users/44f557a478cf4695a1c5155fa513d97b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:00.096665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44f557a478cf4695a1c5155fa513d97b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:00.098746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:00.271939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44f557a478cf4695a1c5155fa513d97b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:00.272559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44f557a478cf4695a1c5155fa513d97b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:00.286187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44f557a478cf4695a1c5155fa513d97b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:00.356597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44f557a478cf4695a1c5155fa513d97b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:00.364026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44f557a478cf4695a1c5155fa513d97b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:00.371178 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1120/2224] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:17:00 2022] DELETE /identity/v3/users/44f557a478cf4695a1c5155fa513d97b => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:00.375411 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e0501-0f6b-4c11-a78c-359fa43f61f9 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:00.378240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e0501-0f6b-4c11-a78c-359fa43f61f9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:00.378240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695e0501-0f6b-4c11-a78c-359fa43f61f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:00.395937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-695e0501-0f6b-4c11-a78c-359fa43f61f9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:00.397144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-695e0501-0f6b-4c11-a78c-359fa43f61f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:00.447307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:00.463972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1105/2225] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:00.472799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:00.474223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:00.536119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ac1a7e-f07f-4f02-ac37-c23c57752aa7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:00.540440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f201788-2349-4839-9162-a38ed7e86f32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:00.540972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f201788-2349-4839-9162-a38ed7e86f32 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:00.541250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f201788-2349-4839-9162-a38ed7e86f32 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:00.541523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f201788-2349-4839-9162-a38ed7e86f32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:00.552740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f201788-2349-4839-9162-a38ed7e86f32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:00.554302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f201788-2349-4839-9162-a38ed7e86f32 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:00.564112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1121/2226] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:17:00.980113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695e0501-0f6b-4c11-a78c-359fa43f61f9 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:00.981685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-695e0501-0f6b-4c11-a78c-359fa43f61f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:00.993701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695e0501-0f6b-4c11-a78c-359fa43f61f9 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:00.999924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:01.000566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:01.000844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:01.001116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] PATH_INFO: `/v3/projects/0cd491659d62499789879d73a1c09dc3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:01.002324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] RBAC: Authorizing `identity:delete_project(project_id=0cd491659d62499789879d73a1c09dc3)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:01.004282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:01.105213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0cd491659d62499789879d73a1c09dc3', 'request_id': 'req-fdc76b32-55f8-4c19-b4db-da22afa9b86d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.155719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0cd491659d62499789879d73a1c09dc3', 'request_id': 'req-fdc76b32-55f8-4c19-b4db-da22afa9b86d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.161532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0cd491659d62499789879d73a1c09dc3', 'request_id': 'req-fdc76b32-55f8-4c19-b4db-da22afa9b86d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.172419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] The token cache is being invalidate because project 0cd491659d62499789879d73a1c09dc3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:17:01.172706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.173326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.174826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1106/2227] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:17:00 2022] DELETE /identity/v3/projects/0cd491659d62499789879d73a1c09dc3 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:01.178886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f201788-2349-4839-9162-a38ed7e86f32 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:01.189044 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f201788-2349-4839-9162-a38ed7e86f32 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:01.236506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:01.245489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:01.246674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:01.246946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:01.247220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] PATH_INFO: `/v3/users/276e6f4c4e62466db82a2722be259c4e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:01.248350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=276e6f4c4e62466db82a2722be259c4e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:01.250302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:01.490904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '276e6f4c4e62466db82a2722be259c4e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.491809 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '276e6f4c4e62466db82a2722be259c4e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.505106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '276e6f4c4e62466db82a2722be259c4e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.649170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '276e6f4c4e62466db82a2722be259c4e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.657991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '276e6f4c4e62466db82a2722be259c4e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:01.663808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1122/2228] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:17:01 2022] DELETE /identity/v3/users/276e6f4c4e62466db82a2722be259c4e => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:01.850282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:01.851704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:01.867118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc76b32-55f8-4c19-b4db-da22afa9b86d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:01.931836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:01.932683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:01.933097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:01.933622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] PATH_INFO: `/v3/users/b9cb460463b44f89b391e145746190c0` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:01.934816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9cb460463b44f89b391e145746190c0)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:01.936820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.100457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.102500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:02.116120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6713f278-f8f6-433c-9e97-23f7752fff44 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.120956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.121911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.122256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.122645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] PATH_INFO: `/v3/projects/523140ae5d6e48858c9bc29e80fc5b04` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.124003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] RBAC: Authorizing `identity:delete_project(project_id=523140ae5d6e48858c9bc29e80fc5b04)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:02.126226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.153338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9cb460463b44f89b391e145746190c0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.165244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9cb460463b44f89b391e145746190c0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.166023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9cb460463b44f89b391e145746190c0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.249745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '523140ae5d6e48858c9bc29e80fc5b04', 'request_id': 'req-eed920f0-26e0-411f-84bb-2c9a401b6331'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.255256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '523140ae5d6e48858c9bc29e80fc5b04', 'request_id': 'req-eed920f0-26e0-411f-84bb-2c9a401b6331'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.259191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9cb460463b44f89b391e145746190c0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.262246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:02.267770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0523bcc9-cae5-46c7-aa22-3afa60c33ae7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9cb460463b44f89b391e145746190c0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.273757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1107/2229] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:17:01 2022] DELETE /identity/v3/users/b9cb460463b44f89b391e145746190c0 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:02.278743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c66deb6-7e6a-4432-804b-6a8beac92b22 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.279133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c66deb6-7e6a-4432-804b-6a8beac92b22 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.279424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c66deb6-7e6a-4432-804b-6a8beac92b22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.299511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4c66deb6-7e6a-4432-804b-6a8beac92b22 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:02.300743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4c66deb6-7e6a-4432-804b-6a8beac92b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:02.344483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '523140ae5d6e48858c9bc29e80fc5b04', 'request_id': 'req-eed920f0-26e0-411f-84bb-2c9a401b6331'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.356058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] The token cache is being invalidate because project 523140ae5d6e48858c9bc29e80fc5b04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:17:02.356348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.356967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eed920f0-26e0-411f-84bb-2c9a401b6331 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.358744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1123/2230] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:17:02 2022] DELETE /identity/v3/projects/523140ae5d6e48858c9bc29e80fc5b04 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:17:02.360556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1108/2231] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:17:02.363153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2351c0c5-b66d-472e-994c-3c7590664208 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.363461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2351c0c5-b66d-472e-994c-3c7590664208 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.363730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2351c0c5-b66d-472e-994c-3c7590664208 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.370309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c66deb6-7e6a-4432-804b-6a8beac92b22 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.372268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4c66deb6-7e6a-4432-804b-6a8beac92b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:02.385071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2351c0c5-b66d-472e-994c-3c7590664208 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:02.386474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2351c0c5-b66d-472e-994c-3c7590664208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:02.394853 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1124/2232] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:02.402429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee14db-cdba-4956-851c-da8b7c1de648 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.402716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee14db-cdba-4956-851c-da8b7c1de648 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.402991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ee14db-cdba-4956-851c-da8b7c1de648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.423258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f3ee14db-cdba-4956-851c-da8b7c1de648 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:02.424457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f3ee14db-cdba-4956-851c-da8b7c1de648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:02.430859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c66deb6-7e6a-4432-804b-6a8beac92b22 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.432506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1125/2233] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:17:02.435366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.435981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.436267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.436543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.437305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ee14db-cdba-4956-851c-da8b7c1de648 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.438447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:02.446563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ee14db-cdba-4956-851c-da8b7c1de648 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.451026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:02.452181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.452461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.452783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.453065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.453336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.454034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:17:02.454337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:02.455926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.462195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1109/2234] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:17:02.468263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1126/2235] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:17:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:17:02.472586 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.482347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff053f3-fc88-4a8f-b103-54a1140005bf service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.487277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.487885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.488165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.488441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.489064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:02.491067 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.598260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1110/2236] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:17:02 2022] POST /identity/v3/projects => generated 381 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:02.602424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.612781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a72077-f2cf-4f9f-906a-d86ebeac8bd7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.641803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:02.672337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d474f65b-f6b1-44db-8932-b177c8f77f73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.672944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d474f65b-f6b1-44db-8932-b177c8f77f73 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.674199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d474f65b-f6b1-44db-8932-b177c8f77f73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.674199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d474f65b-f6b1-44db-8932-b177c8f77f73 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.674199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d474f65b-f6b1-44db-8932-b177c8f77f73 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:02.676067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d474f65b-f6b1-44db-8932-b177c8f77f73 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.700964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.702840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:02.715498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f12f0f-a572-44d8-9628-7d1e14e805e0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.720475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.721152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.721388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.721706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] PATH_INFO: `/v3/projects/1ba0701fef4648c8b434d92a5588afd9` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.722855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ba0701fef4648c8b434d92a5588afd9)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:02.726016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.784793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1127/2237] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:17:02 2022] POST /identity/v3/users => generated 291 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:02.788745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d474f65b-f6b1-44db-8932-b177c8f77f73 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.798592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d474f65b-f6b1-44db-8932-b177c8f77f73 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.803201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6f0c15-59aa-429d-bc91-cba4e2db7c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.803768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6f0c15-59aa-429d-bc91-cba4e2db7c2c admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.804048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6f0c15-59aa-429d-bc91-cba4e2db7c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.804322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6f0c15-59aa-429d-bc91-cba4e2db7c2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.804948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6f0c15-59aa-429d-bc91-cba4e2db7c2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:02.806707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6f0c15-59aa-429d-bc91-cba4e2db7c2c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.817015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1128/2238] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:17:02.820770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6f0c15-59aa-429d-bc91-cba4e2db7c2c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.829826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6f0c15-59aa-429d-bc91-cba4e2db7c2c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.834468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c6d5c9-e7e7-42fa-a914-94eae1d94b07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.835017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c6d5c9-e7e7-42fa-a914-94eae1d94b07 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.835384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c6d5c9-e7e7-42fa-a914-94eae1d94b07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.835657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c6d5c9-e7e7-42fa-a914-94eae1d94b07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.836269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c6d5c9-e7e7-42fa-a914-94eae1d94b07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:02.837881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c6d5c9-e7e7-42fa-a914-94eae1d94b07 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:02.850984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1129/2239] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:02 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:17:02.851923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c6d5c9-e7e7-42fa-a914-94eae1d94b07 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:02.856782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ba0701fef4648c8b434d92a5588afd9', 'request_id': 'req-770eda62-1c02-4448-8067-06228edf3739'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.862257 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c6d5c9-e7e7-42fa-a914-94eae1d94b07 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:02.918444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:02.919112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:02.919415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:02.919713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] PATH_INFO: `/v3/projects/47cb887ee6f74db481d48273a34a1fac/users/79f89dd51cb545c5ba1b2a7496ea41ff/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:02.927217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ba0701fef4648c8b434d92a5588afd9', 'request_id': 'req-770eda62-1c02-4448-8067-06228edf3739'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.934503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ba0701fef4648c8b434d92a5588afd9', 'request_id': 'req-770eda62-1c02-4448-8067-06228edf3739'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.936511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:02.947080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] The token cache is being invalidate because project 1ba0701fef4648c8b434d92a5588afd9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:17:02.947451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.948078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-770eda62-1c02-4448-8067-06228edf3739 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:02.949671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1111/2240] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:17:02 2022] DELETE /identity/v3/projects/1ba0701fef4648c8b434d92a5588afd9 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:17:03.067043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] RBAC: Authorizing `identity:create_grant(project_id=47cb887ee6f74db481d48273a34a1fac, user_id=79f89dd51cb545c5ba1b2a7496ea41ff, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:03.069889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:03.157002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1130/2241] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:17:02 2022] PUT /identity/v3/projects/47cb887ee6f74db481d48273a34a1fac/users/79f89dd51cb545c5ba1b2a7496ea41ff/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:17:03.167237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc7b56-38cd-47a2-a8ff-4c4117f26e70 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:03.167523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc7b56-38cd-47a2-a8ff-4c4117f26e70 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:03.168414 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc7b56-38cd-47a2-a8ff-4c4117f26e70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:03.196563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2dcc7b56-38cd-47a2-a8ff-4c4117f26e70 None None] MFA Rules not processed for user `79f89dd51cb545c5ba1b2a7496ea41ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:03.197855 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2dcc7b56-38cd-47a2-a8ff-4c4117f26e70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:03.261129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1112/2242] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:03 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:03.265644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:03.267321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:03.332619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109e37c8-d5a4-40be-9ca0-59d755e7d39c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:03.337198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e484c808-a97f-4426-8bcc-152d4b0d7c29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:03.337825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e484c808-a97f-4426-8bcc-152d4b0d7c29 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:03.338106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e484c808-a97f-4426-8bcc-152d4b0d7c29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:03.338376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e484c808-a97f-4426-8bcc-152d4b0d7c29 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:03.339030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e484c808-a97f-4426-8bcc-152d4b0d7c29 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:03.341419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e484c808-a97f-4426-8bcc-152d4b0d7c29 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:03.484155 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1131/2243] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:17:03 2022] POST /identity/v3/projects => generated 381 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:17:03.488297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcc7b56-38cd-47a2-a8ff-4c4117f26e70 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:03.499188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcc7b56-38cd-47a2-a8ff-4c4117f26e70 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:03.554968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a24075e-9095-41c1-87b6-e38e73d2e0c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:03.555603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a24075e-9095-41c1-87b6-e38e73d2e0c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:03.555973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a24075e-9095-41c1-87b6-e38e73d2e0c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:03.556233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a24075e-9095-41c1-87b6-e38e73d2e0c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:03.556868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a24075e-9095-41c1-87b6-e38e73d2e0c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:03.558852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a24075e-9095-41c1-87b6-e38e73d2e0c0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:03.693349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1113/2244] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:17:03 2022] POST /identity/v3/users => generated 297 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:03.697410 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e484c808-a97f-4426-8bcc-152d4b0d7c29 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:03.704509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:03.708364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e484c808-a97f-4426-8bcc-152d4b0d7c29 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:03.712842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070b9e3e-b8fe-47e9-9418-a40649431f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:03.713518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070b9e3e-b8fe-47e9-9418-a40649431f96 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:03.713823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070b9e3e-b8fe-47e9-9418-a40649431f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:03.714090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070b9e3e-b8fe-47e9-9418-a40649431f96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:03.714750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070b9e3e-b8fe-47e9-9418-a40649431f96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:03.717002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070b9e3e-b8fe-47e9-9418-a40649431f96 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:03.726938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1132/2245] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:17:03.730947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a24075e-9095-41c1-87b6-e38e73d2e0c0 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:03.740676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a24075e-9095-41c1-87b6-e38e73d2e0c0 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:03.745117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a165a52-5235-4cd9-bf10-153cf810a5bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:03.745793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a165a52-5235-4cd9-bf10-153cf810a5bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:03.746070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a165a52-5235-4cd9-bf10-153cf810a5bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:03.746339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a165a52-5235-4cd9-bf10-153cf810a5bf admin admin] PATH_INFO: `/v3/projects/f58df0ea291e474ba4887e19747f0bf3/users/201ebf02ccf744e99e65d9b4935edab2/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:03.761108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a165a52-5235-4cd9-bf10-153cf810a5bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=f58df0ea291e474ba4887e19747f0bf3, user_id=201ebf02ccf744e99e65d9b4935edab2, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:03.764540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a165a52-5235-4cd9-bf10-153cf810a5bf admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:03.863990 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1114/2246] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:17:03 2022] PUT /identity/v3/projects/f58df0ea291e474ba4887e19747f0bf3/users/201ebf02ccf744e99e65d9b4935edab2/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:03.868001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070b9e3e-b8fe-47e9-9418-a40649431f96 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:03.878451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070b9e3e-b8fe-47e9-9418-a40649431f96 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:03.932669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eeab60-75d0-4114-8aad-097423cca96b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:03.933256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eeab60-75d0-4114-8aad-097423cca96b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:03.933531 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eeab60-75d0-4114-8aad-097423cca96b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:03.933830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eeab60-75d0-4114-8aad-097423cca96b admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:03.934973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73eeab60-75d0-4114-8aad-097423cca96b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:03.936533 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73eeab60-75d0-4114-8aad-097423cca96b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:03.945870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1133/2247] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:03 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:17:03.949943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a165a52-5235-4cd9-bf10-153cf810a5bf admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:03.960652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a165a52-5235-4cd9-bf10-153cf810a5bf admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:03.965177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:03.965853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:03.966133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:03.966412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] PATH_INFO: `/v3/projects/f58df0ea291e474ba4887e19747f0bf3/users/201ebf02ccf744e99e65d9b4935edab2/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:03.968593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f58df0ea291e474ba4887e19747f0bf3, user_id=201ebf02ccf744e99e65d9b4935edab2, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:03.971315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:03.981382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1115/2248] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:17:03 2022] PUT /identity/v3/projects/f58df0ea291e474ba4887e19747f0bf3/users/201ebf02ccf744e99e65d9b4935edab2/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:03.992383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f62433c-d27f-459c-992b-e2603d988748 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:03.992637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f62433c-d27f-459c-992b-e2603d988748 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:03.992909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f62433c-d27f-459c-992b-e2603d988748 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:04.020414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2f62433c-d27f-459c-992b-e2603d988748 None None] MFA Rules not processed for user `201ebf02ccf744e99e65d9b4935edab2`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:04.021608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2f62433c-d27f-459c-992b-e2603d988748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:04.087352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1134/2249] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:03 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:04.098118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:04.099810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:04.141844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:04.156523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9a0fac-4fc6-495c-b332-31237a94454f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:04.160259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20270801-5485-40ac-bb79-c3982039fe80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:04.160854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20270801-5485-40ac-bb79-c3982039fe80 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:04.161131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20270801-5485-40ac-bb79-c3982039fe80 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:04.161401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20270801-5485-40ac-bb79-c3982039fe80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:04.170241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20270801-5485-40ac-bb79-c3982039fe80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:04.171759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20270801-5485-40ac-bb79-c3982039fe80 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:04.181484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1116/2250] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:04 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:17:04.282613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f62433c-d27f-459c-992b-e2603d988748 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:04.292081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f62433c-d27f-459c-992b-e2603d988748 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:04.296241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ec249f-3b47-4f44-addb-647dd2cb33c2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:04.296840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ec249f-3b47-4f44-addb-647dd2cb33c2 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:04.297116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ec249f-3b47-4f44-addb-647dd2cb33c2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:04.297387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ec249f-3b47-4f44-addb-647dd2cb33c2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:04.354909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ec249f-3b47-4f44-addb-647dd2cb33c2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:04.356887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ec249f-3b47-4f44-addb-647dd2cb33c2 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:04.365725 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1135/2251] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:04 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:17:08.475600 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20270801-5485-40ac-bb79-c3982039fe80 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:08.536705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20270801-5485-40ac-bb79-c3982039fe80 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:08.541362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:08.542024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:08.542305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:08.542575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] PATH_INFO: `/v3/users/79f89dd51cb545c5ba1b2a7496ea41ff` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:08.544307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] RBAC: Authorizing `identity:delete_user(user_id=79f89dd51cb545c5ba1b2a7496ea41ff)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:08.546265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:08.752820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79f89dd51cb545c5ba1b2a7496ea41ff'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:08.764760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79f89dd51cb545c5ba1b2a7496ea41ff'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:08.765489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79f89dd51cb545c5ba1b2a7496ea41ff'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:08.870206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79f89dd51cb545c5ba1b2a7496ea41ff'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:08.878479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79f89dd51cb545c5ba1b2a7496ea41ff'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:08.884399 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1117/2252] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:17:08 2022] DELETE /identity/v3/users/79f89dd51cb545c5ba1b2a7496ea41ff => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:08.889010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a17540-322f-42d6-a029-6a98462d7d06 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:08.889522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a17540-322f-42d6-a029-6a98462d7d06 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:08.889656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a17540-322f-42d6-a029-6a98462d7d06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:08.908593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-11a17540-322f-42d6-a029-6a98462d7d06 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:08.910108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-11a17540-322f-42d6-a029-6a98462d7d06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:08.918036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:08.969834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1136/2253] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:08.979607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:08.981632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:09.041857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c3cad6-18e2-447f-adad-759cd4cb3c84 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:09.046054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa541367-669a-4ae3-a973-6daba83b2372 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:09.046664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa541367-669a-4ae3-a973-6daba83b2372 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:09.046944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa541367-669a-4ae3-a973-6daba83b2372 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:09.047216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa541367-669a-4ae3-a973-6daba83b2372 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:09.061788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa541367-669a-4ae3-a973-6daba83b2372 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:09.061788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa541367-669a-4ae3-a973-6daba83b2372 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:09.068849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1118/2254] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:17:09.614492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a17540-322f-42d6-a029-6a98462d7d06 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:09.640303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-11a17540-322f-42d6-a029-6a98462d7d06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:09.640303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a17540-322f-42d6-a029-6a98462d7d06 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:09.640303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:09.641562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:09.641562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:09.641562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] PATH_INFO: `/v3/projects/47cb887ee6f74db481d48273a34a1fac` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:09.641562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47cb887ee6f74db481d48273a34a1fac)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:09.641562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:09.802480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47cb887ee6f74db481d48273a34a1fac', 'request_id': 'req-518173ac-e02c-4654-9a22-f5d76b759d67'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:09.808031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47cb887ee6f74db481d48273a34a1fac', 'request_id': 'req-518173ac-e02c-4654-9a22-f5d76b759d67'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:09.927173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47cb887ee6f74db481d48273a34a1fac', 'request_id': 'req-518173ac-e02c-4654-9a22-f5d76b759d67'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:09.939649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] The token cache is being invalidate because project 47cb887ee6f74db481d48273a34a1fac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:17:09.939937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:09.940558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:09.942060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1137/2255] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:17:09 2022] DELETE /identity/v3/projects/47cb887ee6f74db481d48273a34a1fac => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:09.952228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa541367-669a-4ae3-a973-6daba83b2372 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:09.959284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:09.964652 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa541367-669a-4ae3-a973-6daba83b2372 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:10.019577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:10.020203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:10.020479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:10.020746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] PATH_INFO: `/v3/users/201ebf02ccf744e99e65d9b4935edab2` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:10.021885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=201ebf02ccf744e99e65d9b4935edab2)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:10.023962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:10.223210 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '201ebf02ccf744e99e65d9b4935edab2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:10.242735 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '201ebf02ccf744e99e65d9b4935edab2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:10.243534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '201ebf02ccf744e99e65d9b4935edab2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:10.315730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '201ebf02ccf744e99e65d9b4935edab2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:10.323317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-afd9b412-d25d-4c71-af81-c41f84c09e47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '201ebf02ccf744e99e65d9b4935edab2'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:10.330761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1119/2256] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:17:09 2022] DELETE /identity/v3/users/201ebf02ccf744e99e65d9b4935edab2 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:10.763555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:10.765566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:10.781127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518173ac-e02c-4654-9a22-f5d76b759d67 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:10.839116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:10.839743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:10.840017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:10.840285 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] PATH_INFO: `/v3/projects/f58df0ea291e474ba4887e19747f0bf3` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:10.841975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] RBAC: Authorizing `identity:delete_project(project_id=f58df0ea291e474ba4887e19747f0bf3)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:10.844112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:10.949242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:10.955830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f58df0ea291e474ba4887e19747f0bf3', 'request_id': 'req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:10.961072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f58df0ea291e474ba4887e19747f0bf3', 'request_id': 'req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:11.041405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f58df0ea291e474ba4887e19747f0bf3', 'request_id': 'req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:11.056248 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] The token cache is being invalidate because project f58df0ea291e474ba4887e19747f0bf3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:17:11.056530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:11.057149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-7a83fbfc-5c9b-469e-8fe1-22059627fd8a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:11.058641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1138/2257] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:17:10 2022] DELETE /identity/v3/projects/f58df0ea291e474ba4887e19747f0bf3 => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:11.063576 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cacb55d-87f4-459b-9c73-9f4964a7af33 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.064021 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cacb55d-87f4-459b-9c73-9f4964a7af33 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.064123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cacb55d-87f4-459b-9c73-9f4964a7af33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.083614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-7cacb55d-87f4-459b-9c73-9f4964a7af33 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:11.084793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7cacb55d-87f4-459b-9c73-9f4964a7af33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:11.143119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1120/2258] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:11.150896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ef726a-0acf-4e11-826f-00dd75d36b42 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.151179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ef726a-0acf-4e11-826f-00dd75d36b42 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.151450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ef726a-0acf-4e11-826f-00dd75d36b42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.171337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a8ef726a-0acf-4e11-826f-00dd75d36b42 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:11.172597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a8ef726a-0acf-4e11-826f-00dd75d36b42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:11.180296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1139/2259] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:11.184515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cacb55d-87f4-459b-9c73-9f4964a7af33 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:11.193981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cacb55d-87f4-459b-9c73-9f4964a7af33 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:11.198432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:11.199018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.199294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.199560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.200711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:17:11.201003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:11.202688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:11.213970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1121/2260] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:17:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:17:11.219212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ef726a-0acf-4e11-826f-00dd75d36b42 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:11.228480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ef726a-0acf-4e11-826f-00dd75d36b42 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:11.233279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a8681-993c-451e-b629-0fd51b0628d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:11.233929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a8681-993c-451e-b629-0fd51b0628d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.234206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a8681-993c-451e-b629-0fd51b0628d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.234474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a8681-993c-451e-b629-0fd51b0628d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.235097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838a8681-993c-451e-b629-0fd51b0628d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:11.236972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838a8681-993c-451e-b629-0fd51b0628d3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:11.352093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1140/2261] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:17:11 2022] POST /identity/v3/projects => generated 385 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:11.356557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:11.366731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2da033-f737-45ec-ad25-dba156b6d0b1 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:11.372588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:11.424610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a783f1f5-f019-4a5c-98f7-10c854583852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:11.425249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a783f1f5-f019-4a5c-98f7-10c854583852 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.425499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a783f1f5-f019-4a5c-98f7-10c854583852 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.426390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a783f1f5-f019-4a5c-98f7-10c854583852 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.427036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a783f1f5-f019-4a5c-98f7-10c854583852 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:11.428945 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a783f1f5-f019-4a5c-98f7-10c854583852 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:11.562082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1122/2262] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:17:11 2022] POST /identity/v3/users => generated 293 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:11.566418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a8681-993c-451e-b629-0fd51b0628d3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:11.577023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a8681-993c-451e-b629-0fd51b0628d3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:11.581490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a107741-d5e9-40a6-99b8-278f33a543a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:11.582105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a107741-d5e9-40a6-99b8-278f33a543a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.582406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a107741-d5e9-40a6-99b8-278f33a543a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.582674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a107741-d5e9-40a6-99b8-278f33a543a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.583305 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a107741-d5e9-40a6-99b8-278f33a543a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:11.584866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a107741-d5e9-40a6-99b8-278f33a543a2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:11.595097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1141/2263] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:17:11.609475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a783f1f5-f019-4a5c-98f7-10c854583852 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:11.619538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a783f1f5-f019-4a5c-98f7-10c854583852 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:11.624884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2017df09-66f2-4e4c-a9b8-e05e96b7ee97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:11.625480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2017df09-66f2-4e4c-a9b8-e05e96b7ee97 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.625789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2017df09-66f2-4e4c-a9b8-e05e96b7ee97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.626058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2017df09-66f2-4e4c-a9b8-e05e96b7ee97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.626719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2017df09-66f2-4e4c-a9b8-e05e96b7ee97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:11.628318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2017df09-66f2-4e4c-a9b8-e05e96b7ee97 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:11.641583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1123/2264] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:17:11.645370 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a107741-d5e9-40a6-99b8-278f33a543a2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:11.655693 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a107741-d5e9-40a6-99b8-278f33a543a2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:11.660142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f61c68-69d4-47ab-9de6-f8e85c4d4f14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:11.660787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f61c68-69d4-47ab-9de6-f8e85c4d4f14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.661061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f61c68-69d4-47ab-9de6-f8e85c4d4f14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.661326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f61c68-69d4-47ab-9de6-f8e85c4d4f14 admin admin] PATH_INFO: `/v3/projects/7cfe0085ed8f4046a4b0996866efbb3a/users/6fc499b95a5a4db4a3f79b5001ef4993/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.676082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f61c68-69d4-47ab-9de6-f8e85c4d4f14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cfe0085ed8f4046a4b0996866efbb3a, user_id=6fc499b95a5a4db4a3f79b5001ef4993, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:11.678730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f61c68-69d4-47ab-9de6-f8e85c4d4f14 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:11.767479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1142/2265] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:17:11 2022] PUT /identity/v3/projects/7cfe0085ed8f4046a4b0996866efbb3a/users/6fc499b95a5a4db4a3f79b5001ef4993/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:11.777898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf19e6-808c-45f1-87bb-7f5b854412a4 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.778190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf19e6-808c-45f1-87bb-7f5b854412a4 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.778702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf19e6-808c-45f1-87bb-7f5b854412a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.806077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3ddf19e6-808c-45f1-87bb-7f5b854412a4 None None] MFA Rules not processed for user `6fc499b95a5a4db4a3f79b5001ef4993`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:11.807598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3ddf19e6-808c-45f1-87bb-7f5b854412a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:11.869542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1124/2266] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:11 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:11.874474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f61c68-69d4-47ab-9de6-f8e85c4d4f14 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:11.934999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f61c68-69d4-47ab-9de6-f8e85c4d4f14 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:11.939507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d98415a-efcd-4403-8240-f5c2f39515f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:11.940101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d98415a-efcd-4403-8240-f5c2f39515f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:11.940378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d98415a-efcd-4403-8240-f5c2f39515f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:11.940649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d98415a-efcd-4403-8240-f5c2f39515f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:11.941269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d98415a-efcd-4403-8240-f5c2f39515f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:11.943181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d98415a-efcd-4403-8240-f5c2f39515f5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:12.036352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1143/2267] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:17:11 2022] POST /identity/v3/projects => generated 383 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:12.040407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf19e6-808c-45f1-87bb-7f5b854412a4 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:12.051694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf19e6-808c-45f1-87bb-7f5b854412a4 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:12.108370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965dfde1-cf20-466a-82a2-774e4bb1d21f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:12.108981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965dfde1-cf20-466a-82a2-774e4bb1d21f admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.109240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965dfde1-cf20-466a-82a2-774e4bb1d21f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.109514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965dfde1-cf20-466a-82a2-774e4bb1d21f admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.110160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965dfde1-cf20-466a-82a2-774e4bb1d21f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:12.112076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965dfde1-cf20-466a-82a2-774e4bb1d21f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:12.195352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1125/2268] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:17:12 2022] POST /identity/v3/users => generated 292 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:12.199534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d98415a-efcd-4403-8240-f5c2f39515f5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:12.209577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d98415a-efcd-4403-8240-f5c2f39515f5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:12.214238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9cddb6-cfbd-4333-ba5e-e3c95c774f9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:12.214827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9cddb6-cfbd-4333-ba5e-e3c95c774f9f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.215107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9cddb6-cfbd-4333-ba5e-e3c95c774f9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.215885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9cddb6-cfbd-4333-ba5e-e3c95c774f9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.216520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9cddb6-cfbd-4333-ba5e-e3c95c774f9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:12.218413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9cddb6-cfbd-4333-ba5e-e3c95c774f9f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:12.229018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1144/2269] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:17:12.233581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965dfde1-cf20-466a-82a2-774e4bb1d21f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:12.243626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965dfde1-cf20-466a-82a2-774e4bb1d21f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:12.251875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dccc5ec-ffe4-4819-93ae-70359693c356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:12.251875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccc5ec-ffe4-4819-93ae-70359693c356 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.251875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccc5ec-ffe4-4819-93ae-70359693c356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.251875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dccc5ec-ffe4-4819-93ae-70359693c356 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.252261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dccc5ec-ffe4-4819-93ae-70359693c356 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:12.252261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dccc5ec-ffe4-4819-93ae-70359693c356 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:12.253051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:12.262067 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1126/2270] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:17:12.266153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9cddb6-cfbd-4333-ba5e-e3c95c774f9f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:12.275794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9cddb6-cfbd-4333-ba5e-e3c95c774f9f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:12.280938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda937c6-e780-4e0d-985d-c4b88ebb9e54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:12.281571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda937c6-e780-4e0d-985d-c4b88ebb9e54 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.281877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda937c6-e780-4e0d-985d-c4b88ebb9e54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.282148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda937c6-e780-4e0d-985d-c4b88ebb9e54 admin admin] PATH_INFO: `/v3/projects/4a1a459d9e4b449d803f46b786544683/users/60532662d21c4982ab56f5ec0a4cdcdb/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.297378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda937c6-e780-4e0d-985d-c4b88ebb9e54 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a1a459d9e4b449d803f46b786544683, user_id=60532662d21c4982ab56f5ec0a4cdcdb, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:12.300142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda937c6-e780-4e0d-985d-c4b88ebb9e54 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:12.378149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1145/2271] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:17:12 2022] PUT /identity/v3/projects/4a1a459d9e4b449d803f46b786544683/users/60532662d21c4982ab56f5ec0a4cdcdb/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:12.444678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8dcfcf-3f14-49cf-aac3-bb3fb32387aa None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.444678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8dcfcf-3f14-49cf-aac3-bb3fb32387aa None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.444887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8dcfcf-3f14-49cf-aac3-bb3fb32387aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.474572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5d8dcfcf-3f14-49cf-aac3-bb3fb32387aa None None] MFA Rules not processed for user `60532662d21c4982ab56f5ec0a4cdcdb`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:12.475784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5d8dcfcf-3f14-49cf-aac3-bb3fb32387aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:12.541684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1127/2272] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:12.546288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda937c6-e780-4e0d-985d-c4b88ebb9e54 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:12.561368 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:12.611022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda937c6-e780-4e0d-985d-c4b88ebb9e54 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:12.615711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e6d97-2b9f-4f96-b2e3-2b68140d8bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:12.616311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e6d97-2b9f-4f96-b2e3-2b68140d8bd6 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.616585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e6d97-2b9f-4f96-b2e3-2b68140d8bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.616858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e6d97-2b9f-4f96-b2e3-2b68140d8bd6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.617484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e6d97-2b9f-4f96-b2e3-2b68140d8bd6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:12.619388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e6d97-2b9f-4f96-b2e3-2b68140d8bd6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:12.688164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1146/2273] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:17:12 2022] POST /identity/v3/projects => generated 383 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:12.692168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8dcfcf-3f14-49cf-aac3-bb3fb32387aa None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:12.704015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8dcfcf-3f14-49cf-aac3-bb3fb32387aa None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:12.770265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b65987b-5c1e-4e84-ad4f-56888b6bff57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:12.770886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b65987b-5c1e-4e84-ad4f-56888b6bff57 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.771172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b65987b-5c1e-4e84-ad4f-56888b6bff57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.771473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b65987b-5c1e-4e84-ad4f-56888b6bff57 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.772104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b65987b-5c1e-4e84-ad4f-56888b6bff57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:12.774109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b65987b-5c1e-4e84-ad4f-56888b6bff57 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:12.876046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1128/2274] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:17:12 2022] POST /identity/v3/users => generated 298 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:17:12.879945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e6d97-2b9f-4f96-b2e3-2b68140d8bd6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:12.890258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e6d97-2b9f-4f96-b2e3-2b68140d8bd6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:12.894906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332c39ba-355e-400f-acba-5a9000634fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:12.895500 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332c39ba-355e-400f-acba-5a9000634fc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.895781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332c39ba-355e-400f-acba-5a9000634fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.896056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332c39ba-355e-400f-acba-5a9000634fc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.896803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332c39ba-355e-400f-acba-5a9000634fc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:12.898444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332c39ba-355e-400f-acba-5a9000634fc6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:12.909383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1147/2275] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:17:12.913429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b65987b-5c1e-4e84-ad4f-56888b6bff57 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:12.924431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b65987b-5c1e-4e84-ad4f-56888b6bff57 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:12.929065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9952fa-74d2-418f-9267-0caf26375a79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:12.929778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9952fa-74d2-418f-9267-0caf26375a79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:12.930061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9952fa-74d2-418f-9267-0caf26375a79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:12.930336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9952fa-74d2-418f-9267-0caf26375a79 admin admin] PATH_INFO: `/v3/projects/fccf873e106245279d1269618bfa41b2/users/0c90de5d254d4d458422bb3f461531eb/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:12.944697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9952fa-74d2-418f-9267-0caf26375a79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fccf873e106245279d1269618bfa41b2, user_id=0c90de5d254d4d458422bb3f461531eb, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:12.947395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9952fa-74d2-418f-9267-0caf26375a79 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:13.013218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1129/2276] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:17:12 2022] PUT /identity/v3/projects/fccf873e106245279d1269618bfa41b2/users/0c90de5d254d4d458422bb3f461531eb/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:13.016998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332c39ba-355e-400f-acba-5a9000634fc6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:13.026909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332c39ba-355e-400f-acba-5a9000634fc6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:13.082695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07028c46-8cca-4f10-aa74-b37caa16f5d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:13.083327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07028c46-8cca-4f10-aa74-b37caa16f5d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:13.083562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07028c46-8cca-4f10-aa74-b37caa16f5d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:13.083836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07028c46-8cca-4f10-aa74-b37caa16f5d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:13.084469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07028c46-8cca-4f10-aa74-b37caa16f5d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:13.086080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07028c46-8cca-4f10-aa74-b37caa16f5d6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:13.097272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1148/2277] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:17:13 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:17:13.101491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9952fa-74d2-418f-9267-0caf26375a79 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:13.113233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9952fa-74d2-418f-9267-0caf26375a79 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:13.117882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:13.118534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:13.118818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:13.119096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] PATH_INFO: `/v3/projects/fccf873e106245279d1269618bfa41b2/users/0c90de5d254d4d458422bb3f461531eb/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:13.121050 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fccf873e106245279d1269618bfa41b2, user_id=0c90de5d254d4d458422bb3f461531eb, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:13.124135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:13.134402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1130/2278] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:17:13 2022] PUT /identity/v3/projects/fccf873e106245279d1269618bfa41b2/users/0c90de5d254d4d458422bb3f461531eb/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:13.144373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ae6e33-602b-478e-8904-b5a7902594dc None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:13.144665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ae6e33-602b-478e-8904-b5a7902594dc None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:13.144944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ae6e33-602b-478e-8904-b5a7902594dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:13.172530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e3ae6e33-602b-478e-8904-b5a7902594dc None None] MFA Rules not processed for user `0c90de5d254d4d458422bb3f461531eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:13.173758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e3ae6e33-602b-478e-8904-b5a7902594dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:13.240781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1149/2279] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:13 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:13.251952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:13.253685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:13.295422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:13.309955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abac6cb3-4a1f-4c55-ac3a-38eb5700b1c2 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:13.313676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e96e4e-f28e-482c-beba-41c61c44bbe0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:13.314215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e96e4e-f28e-482c-beba-41c61c44bbe0 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:13.314494 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e96e4e-f28e-482c-beba-41c61c44bbe0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:13.314760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e96e4e-f28e-482c-beba-41c61c44bbe0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:13.370991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e96e4e-f28e-482c-beba-41c61c44bbe0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:13.373089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e96e4e-f28e-482c-beba-41c61c44bbe0 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:13.381852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1131/2280] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:13 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:17:16.430739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ae6e33-602b-478e-8904-b5a7902594dc None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:16.440469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ae6e33-602b-478e-8904-b5a7902594dc None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:16.444225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4156eca-9130-46b3-beda-edc080f87c95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:16.444829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4156eca-9130-46b3-beda-edc080f87c95 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:16.445103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4156eca-9130-46b3-beda-edc080f87c95 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:16.445370 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4156eca-9130-46b3-beda-edc080f87c95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:16.451900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:16.504888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4156eca-9130-46b3-beda-edc080f87c95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:16.506358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4156eca-9130-46b3-beda-edc080f87c95 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:16.516308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1150/2281] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:17:17.876656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e96e4e-f28e-482c-beba-41c61c44bbe0 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:17.884961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e96e4e-f28e-482c-beba-41c61c44bbe0 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:17.889012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503dea08-a3cb-4bdf-89bf-a8ece2f4bce6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:17.889607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503dea08-a3cb-4bdf-89bf-a8ece2f4bce6 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:17.889909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503dea08-a3cb-4bdf-89bf-a8ece2f4bce6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:17.890177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503dea08-a3cb-4bdf-89bf-a8ece2f4bce6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:17.898857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503dea08-a3cb-4bdf-89bf-a8ece2f4bce6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:17.900266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503dea08-a3cb-4bdf-89bf-a8ece2f4bce6 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:17.909992 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1132/2282] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:17 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:17:21.871895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6fc97-593c-4e54-a5d3-f0487dcdcc8e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:21.874153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6fc97-593c-4e54-a5d3-f0487dcdcc8e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:21.874153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f6fc97-593c-4e54-a5d3-f0487dcdcc8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:21.890752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-57f6fc97-593c-4e54-a5d3-f0487dcdcc8e None None] MFA Rules not processed for user `ec39d995899c4d43bb4f8ffd9e6d2ef3`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:21.891895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-57f6fc97-593c-4e54-a5d3-f0487dcdcc8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:21.946097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1151/2283] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:22.778145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503dea08-a3cb-4bdf-89bf-a8ece2f4bce6 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:22.841474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503dea08-a3cb-4bdf-89bf-a8ece2f4bce6 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:22.846377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:22.847005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:22.847285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:22.847558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] PATH_INFO: `/v3/users/6fc499b95a5a4db4a3f79b5001ef4993` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:22.848670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6fc499b95a5a4db4a3f79b5001ef4993)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:22.850604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:23.118570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6fc499b95a5a4db4a3f79b5001ef4993'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:23.136039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6fc499b95a5a4db4a3f79b5001ef4993'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:23.136879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6fc499b95a5a4db4a3f79b5001ef4993'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:23.219272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6fc499b95a5a4db4a3f79b5001ef4993'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:23.231574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6fc499b95a5a4db4a3f79b5001ef4993'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:23.231574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:23.234642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1133/2284] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:17:22 2022] DELETE /identity/v3/users/6fc499b95a5a4db4a3f79b5001ef4993 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:23.239078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7f7fe3-11e5-4413-9982-b6e421718055 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:23.239520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7f7fe3-11e5-4413-9982-b6e421718055 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:23.239611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7f7fe3-11e5-4413-9982-b6e421718055 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:23.258950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3a7f7fe3-11e5-4413-9982-b6e421718055 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:23.260154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3a7f7fe3-11e5-4413-9982-b6e421718055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:23.319015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1152/2285] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:17:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:23.328954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:23.331163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:23.407710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27de97ef-4518-4464-bf39-b7d2aa6a90d1 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:23.415248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea54127-4da3-4650-9b7f-ecb8e3338dc3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:23.416963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea54127-4da3-4650-9b7f-ecb8e3338dc3 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:23.417419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea54127-4da3-4650-9b7f-ecb8e3338dc3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:23.418069 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea54127-4da3-4650-9b7f-ecb8e3338dc3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:23.431228 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea54127-4da3-4650-9b7f-ecb8e3338dc3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:23.433338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea54127-4da3-4650-9b7f-ecb8e3338dc3 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:23.450109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1134/2286] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:17:24.061695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7f7fe3-11e5-4413-9982-b6e421718055 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:24.062108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3a7f7fe3-11e5-4413-9982-b6e421718055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:24.075676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7f7fe3-11e5-4413-9982-b6e421718055 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:24.080544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:24.081200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:24.081482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:24.081784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] PATH_INFO: `/v3/projects/7cfe0085ed8f4046a4b0996866efbb3a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:24.082942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cfe0085ed8f4046a4b0996866efbb3a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:24.084852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:24.237352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:24.238846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cfe0085ed8f4046a4b0996866efbb3a', 'request_id': 'req-53f9cf4b-d547-4e83-b758-6a3a3401d111'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.244509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cfe0085ed8f4046a4b0996866efbb3a', 'request_id': 'req-53f9cf4b-d547-4e83-b758-6a3a3401d111'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.287442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cfe0085ed8f4046a4b0996866efbb3a', 'request_id': 'req-53f9cf4b-d547-4e83-b758-6a3a3401d111'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.299637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] The token cache is being invalidate because project 7cfe0085ed8f4046a4b0996866efbb3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:17:24.300050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.300755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.302359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1153/2287] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:17:24 2022] DELETE /identity/v3/projects/7cfe0085ed8f4046a4b0996866efbb3a => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:24.306511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea54127-4da3-4650-9b7f-ecb8e3338dc3 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:24.316167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea54127-4da3-4650-9b7f-ecb8e3338dc3 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:24.373291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:24.373957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:24.374233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:24.374503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] PATH_INFO: `/v3/users/60532662d21c4982ab56f5ec0a4cdcdb` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:24.375626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=60532662d21c4982ab56f5ec0a4cdcdb)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:24.377614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:24.588489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '60532662d21c4982ab56f5ec0a4cdcdb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.602029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '60532662d21c4982ab56f5ec0a4cdcdb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.602743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '60532662d21c4982ab56f5ec0a4cdcdb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.681366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '60532662d21c4982ab56f5ec0a4cdcdb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.688955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '60532662d21c4982ab56f5ec0a4cdcdb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:24.695084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1135/2288] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:17:24 2022] DELETE /identity/v3/users/60532662d21c4982ab56f5ec0a4cdcdb => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:25.150938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:25.152621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:25.168926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9cf4b-d547-4e83-b758-6a3a3401d111 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:25.234323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:25.235031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:25.235255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:25.235586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] PATH_INFO: `/v3/projects/4a1a459d9e4b449d803f46b786544683` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:25.237339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a1a459d9e4b449d803f46b786544683)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:25.239385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:25.349708 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a1a459d9e4b449d803f46b786544683', 'request_id': 'req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.355389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a1a459d9e4b449d803f46b786544683', 'request_id': 'req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.395748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a1a459d9e4b449d803f46b786544683', 'request_id': 'req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.406772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] The token cache is being invalidate because project 4a1a459d9e4b449d803f46b786544683 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:17:25.407059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.407679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.409186 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1154/2289] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:17:25 2022] DELETE /identity/v3/projects/4a1a459d9e4b449d803f46b786544683 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:25.413257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:25.414756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:25.428262 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39c1102-fd44-49fe-a460-e93993e075c9 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:25.446696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:25.494483 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:25.495408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:25.495685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:25.495958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] PATH_INFO: `/v3/users/0c90de5d254d4d458422bb3f461531eb` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:25.497071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c90de5d254d4d458422bb3f461531eb)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:25.499026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:25.735080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c90de5d254d4d458422bb3f461531eb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.747851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c90de5d254d4d458422bb3f461531eb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.748644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c90de5d254d4d458422bb3f461531eb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.813026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c90de5d254d4d458422bb3f461531eb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.820728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-00ba8cd9-da62-4007-831f-80dfcf80cac6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c90de5d254d4d458422bb3f461531eb'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:25.826750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1136/2290] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:17:25 2022] DELETE /identity/v3/users/0c90de5d254d4d458422bb3f461531eb => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:26.248828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:26.250329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:26.266078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d5cc70-5e70-4fa3-a7c9-6cc0d3aae8d4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:26.323764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:26.324391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:26.324973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:26.325254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] PATH_INFO: `/v3/projects/fccf873e106245279d1269618bfa41b2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:26.326457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fccf873e106245279d1269618bfa41b2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:26.328367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:26.336887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:26.436939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fccf873e106245279d1269618bfa41b2', 'request_id': 'req-072332d0-ae22-44f9-89c3-c0af58d6c8c9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:26.443014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fccf873e106245279d1269618bfa41b2', 'request_id': 'req-072332d0-ae22-44f9-89c3-c0af58d6c8c9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:26.522083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fccf873e106245279d1269618bfa41b2', 'request_id': 'req-072332d0-ae22-44f9-89c3-c0af58d6c8c9'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:26.534391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] The token cache is being invalidate because project fccf873e106245279d1269618bfa41b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:17:26.534798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:26.535464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:17:26.537024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1155/2291] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:17:26 2022] DELETE /identity/v3/projects/fccf873e106245279d1269618bfa41b2 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:17:32.786758 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9dfbc6-5ab0-4126-8b5e-977045813aaf None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:32.787234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9dfbc6-5ab0-4126-8b5e-977045813aaf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:32.787330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9dfbc6-5ab0-4126-8b5e-977045813aaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:32.806125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1d9dfbc6-5ab0-4126-8b5e-977045813aaf None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:32.807321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1d9dfbc6-5ab0-4126-8b5e-977045813aaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:32.862495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1137/2292] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:17:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:32.884636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:32.886180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:32.971254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072332d0-ae22-44f9-89c3-c0af58d6c8c9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:32.976934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752e84e4-6a6c-41cf-a19c-ab77b5a07db9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:32.977667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e84e4-6a6c-41cf-a19c-ab77b5a07db9 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:32.978050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e84e4-6a6c-41cf-a19c-ab77b5a07db9 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:32.978393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e84e4-6a6c-41cf-a19c-ab77b5a07db9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:32.990779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752e84e4-6a6c-41cf-a19c-ab77b5a07db9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:32.992567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752e84e4-6a6c-41cf-a19c-ab77b5a07db9 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:33.001728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1156/2293] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:17:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:17:42.855312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=69,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:42.998499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=53,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:17:50.826915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc423f-9610-4288-b248-81ad1bc0aa4e None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:50.827432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc423f-9610-4288-b248-81ad1bc0aa4e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:50.827607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc423f-9610-4288-b248-81ad1bc0aa4e None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:50.829279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1138/2294] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:17:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:17:50.835901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2b5a59-537a-40e9-9116-959b22a83379 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:50.836431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2b5a59-537a-40e9-9116-959b22a83379 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:50.836536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2b5a59-537a-40e9-9116-959b22a83379 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:50.857617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1f2b5a59-537a-40e9-9116-959b22a83379 None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:50.858840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1f2b5a59-537a-40e9-9116-959b22a83379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:50.930825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1157/2295] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:17:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:50.942604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dc423f-9610-4288-b248-81ad1bc0aa4e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:50.944282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d9dc423f-9610-4288-b248-81ad1bc0aa4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:51.002682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dc423f-9610-4288-b248-81ad1bc0aa4e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:51.007140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85d236-7bb6-45cb-afb1-fc4f1e6fa51b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6cc8f34f9674a94ba23a7157e44a6f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:51.007759 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85d236-7bb6-45cb-afb1-fc4f1e6fa51b service glance] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:51.008043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85d236-7bb6-45cb-afb1-fc4f1e6fa51b service glance] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:51.008317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85d236-7bb6-45cb-afb1-fc4f1e6fa51b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:51.019182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb85d236-7bb6-45cb-afb1-fc4f1e6fa51b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:51.020627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb85d236-7bb6-45cb-afb1-fc4f1e6fa51b service glance] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:51.031500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1139/2296] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:17:51.254669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d70bcf-0d36-4df0-b1c4-537f49aef5c0 None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:51.254952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d70bcf-0d36-4df0-b1c4-537f49aef5c0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:51.255222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d70bcf-0d36-4df0-b1c4-537f49aef5c0 None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:51.256577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1158/2297] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:17:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:17:51.261335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783cd21a-6604-411a-b764-27340c74187b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:51.261655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783cd21a-6604-411a-b764-27340c74187b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:51.261974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783cd21a-6604-411a-b764-27340c74187b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:51.281912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-783cd21a-6604-411a-b764-27340c74187b None None] MFA Rules not processed for user `9220ba3637a542a382ad1d42ff1ff459`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:51.283159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-783cd21a-6604-411a-b764-27340c74187b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:51.334814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1140/2298] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:17:51 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:51.346694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d70bcf-0d36-4df0-b1c4-537f49aef5c0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:51.348362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e7d70bcf-0d36-4df0-b1c4-537f49aef5c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:51.361091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d70bcf-0d36-4df0-b1c4-537f49aef5c0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:51.365097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ceff0b7-a6ae-43b7-b201-744bb112ac23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:51.365712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceff0b7-a6ae-43b7-b201-744bb112ac23 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:51.365990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceff0b7-a6ae-43b7-b201-744bb112ac23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:51.366258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceff0b7-a6ae-43b7-b201-744bb112ac23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:51.373070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceff0b7-a6ae-43b7-b201-744bb112ac23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:51.374502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceff0b7-a6ae-43b7-b201-744bb112ac23 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:51.382713 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1159/2299] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:17:51.466264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadf06de-106a-41c1-9a7c-405639d008b0 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:51.466552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadf06de-106a-41c1-9a7c-405639d008b0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:51.467617 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadf06de-106a-41c1-9a7c-405639d008b0 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:51.468655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1141/2300] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:17:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:17:51.473937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d4c13-1658-4694-9e79-faca9208a190 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:51.474247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d4c13-1658-4694-9e79-faca9208a190 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:51.474524 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d4c13-1658-4694-9e79-faca9208a190 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:51.495567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f67d4c13-1658-4694-9e79-faca9208a190 None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:51.496804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f67d4c13-1658-4694-9e79-faca9208a190 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:51.507022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1160/2301] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:17:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:51.517562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadf06de-106a-41c1-9a7c-405639d008b0 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:51.528718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadf06de-106a-41c1-9a7c-405639d008b0 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:51.532394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22193e88-ed1d-4475-a450-4e6daa15ad94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:51.532999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22193e88-ed1d-4475-a450-4e6daa15ad94 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:51.533282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22193e88-ed1d-4475-a450-4e6daa15ad94 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:51.533556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22193e88-ed1d-4475-a450-4e6daa15ad94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:51.543733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22193e88-ed1d-4475-a450-4e6daa15ad94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:51.545173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22193e88-ed1d-4475-a450-4e6daa15ad94 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:51.557556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1142/2302] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:17:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:17:55.530552 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62201c00-1815-4dd7-98ac-1323f9818c45 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:55.531050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62201c00-1815-4dd7-98ac-1323f9818c45 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:55.531416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62201c00-1815-4dd7-98ac-1323f9818c45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:55.552126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-62201c00-1815-4dd7-98ac-1323f9818c45 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:17:55.553449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-62201c00-1815-4dd7-98ac-1323f9818c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:17:55.564306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1161/2303] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:17:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:17:55.576976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22193e88-ed1d-4475-a450-4e6daa15ad94 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:17:55.588974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22193e88-ed1d-4475-a450-4e6daa15ad94 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:17:55.594508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c712447-158c-4422-9e1a-e70ab1ee549d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:17:55.595115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c712447-158c-4422-9e1a-e70ab1ee549d service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:17:55.595394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c712447-158c-4422-9e1a-e70ab1ee549d service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:17:55.595687 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c712447-158c-4422-9e1a-e70ab1ee549d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:17:55.604661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c712447-158c-4422-9e1a-e70ab1ee549d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:17:55.606327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c712447-158c-4422-9e1a-e70ab1ee549d service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:17:55.614995 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1143/2304] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:17:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:18:05.546902 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=43 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:18:05.611612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=61 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:18:22.690785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62201c00-1815-4dd7-98ac-1323f9818c45 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:18:22.692269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-62201c00-1815-4dd7-98ac-1323f9818c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:18:22.703526 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62201c00-1815-4dd7-98ac-1323f9818c45 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:18:22.707656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:18:22.708250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:18:22.708529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:18:22.708802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:18:22.710594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:18:22.723828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:18:22.725269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:18:22.735746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1162/2305] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:18:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:18:24.134369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885398b2-79e2-4261-aa4d-09795649b4ff None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:18:24.134859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885398b2-79e2-4261-aa4d-09795649b4ff None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:18:24.135128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885398b2-79e2-4261-aa4d-09795649b4ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:18:24.154546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-885398b2-79e2-4261-aa4d-09795649b4ff None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:18:24.155791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-885398b2-79e2-4261-aa4d-09795649b4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:18:24.162820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1144/2306] 10.222.0.16 () {58 vars in 967 bytes} [Thu Aug 25 05:18:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:18:24.174896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:18:24.185788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee3d64a-84cb-4eb7-9e51-aad4d6b99bbd service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:18:24.191165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ce98b0-0480-49f3-a25c-c845dcec6ac2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:18:24.191746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ce98b0-0480-49f3-a25c-c845dcec6ac2 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:18:24.192022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ce98b0-0480-49f3-a25c-c845dcec6ac2 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:18:24.192293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ce98b0-0480-49f3-a25c-c845dcec6ac2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:18:24.200376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ce98b0-0480-49f3-a25c-c845dcec6ac2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:18:24.201993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ce98b0-0480-49f3-a25c-c845dcec6ac2 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:18:24.210403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1163/2307] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:18:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:18:24.831743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885398b2-79e2-4261-aa4d-09795649b4ff None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:18:24.841053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885398b2-79e2-4261-aa4d-09795649b4ff None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:18:24.844816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:18:24.845409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:18:24.845725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:18:24.846005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:18:24.848185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:18:24.861194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:18:24.864916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:18:24.874265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1145/2308] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:18:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:18:34.207761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=12 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:18:34.337709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a19b16-4e76-4972-b894-7382797b81f3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:18:34.337992 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a19b16-4e76-4972-b894-7382797b81f3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:18:34.338261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a19b16-4e76-4972-b894-7382797b81f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:18:34.358289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f3a19b16-4e76-4972-b894-7382797b81f3 None None] MFA Rules not processed for user `b6ca9962541744bf9d67cc3e05721413`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:18:34.359467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f3a19b16-4e76-4972-b894-7382797b81f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:18:34.416074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1164/2309] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:18:34 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:18:34.869422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=9 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:18:44.409232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=25 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:18:59.826964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:18:59.828485 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:18:59.842368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8effe49-c91b-4abd-ad50-8c62a27dbb09 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:18:59.848556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:18:59.849416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:18:59.849731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:18:59.850004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:18:59.851672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:18:59.911236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:18:59.912932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:18:59.922311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1146/2310] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:18:59 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:09.264683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a19b16-4e76-4972-b894-7382797b81f3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:09.277854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a19b16-4e76-4972-b894-7382797b81f3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:09.283330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:09.283932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:09.284211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:09.284484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:09.286241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:09.343956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:09.345839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:09.361461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1165/2311] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:09 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:09.918124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=28 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:19:12.622394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:12.634031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4386adb0-90f7-4fdf-babb-125b7192df14 service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:12.640283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:12.640974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:12.641324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:12.641693 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:12.644089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:12.656892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:12.658462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:12.670272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1147/2312] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:15.215563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:15.226337 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e954d-c938-4a52-98e8-cd3b7e442c37 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:15.231904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80166d97-6e6d-4dbd-a370-79d19b78294b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:15.232520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80166d97-6e6d-4dbd-a370-79d19b78294b service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:15.232929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80166d97-6e6d-4dbd-a370-79d19b78294b service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:15.233205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80166d97-6e6d-4dbd-a370-79d19b78294b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:15.235002 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-80166d97-6e6d-4dbd-a370-79d19b78294b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:15.244760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80166d97-6e6d-4dbd-a370-79d19b78294b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:15.246490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80166d97-6e6d-4dbd-a370-79d19b78294b service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:15.254908 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1166/2313] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:22.666452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:19:25.218332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:25.231609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d16ca-f694-4938-82ea-bb2a458bed4d service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:25.238286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:25.238902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:25.239182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:25.239456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:25.241200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:25.250800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=34 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:19:25.299028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:25.301357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:25.310123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1148/2314] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:25 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:27.293972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb934ca-9a71-44e6-a40d-7aec56788fde None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:27.294441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb934ca-9a71-44e6-a40d-7aec56788fde None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:27.294534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb934ca-9a71-44e6-a40d-7aec56788fde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:27.313477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6eb934ca-9a71-44e6-a40d-7aec56788fde None None] MFA Rules not processed for user `ec39d995899c4d43bb4f8ffd9e6d2ef3`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:19:27.314680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6eb934ca-9a71-44e6-a40d-7aec56788fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:27.321283 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1167/2315] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:19:27 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:19:27.631246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:27.640273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bba9a-27fa-4fdf-9679-572a630c0584 service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:27.644474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:27.645065 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:27.645342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:27.645629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:27.647342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:27.656689 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:27.658181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:27.666160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1149/2316] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:27 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:28.030968 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb934ca-9a71-44e6-a40d-7aec56788fde None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:28.032381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6eb934ca-9a71-44e6-a40d-7aec56788fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:28.112977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb934ca-9a71-44e6-a40d-7aec56788fde None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:28.117988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b279a35-6d17-4e86-95c6-4c81d139eb8a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:28.118597 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b279a35-6d17-4e86-95c6-4c81d139eb8a service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:28.118954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b279a35-6d17-4e86-95c6-4c81d139eb8a service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:28.119183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b279a35-6d17-4e86-95c6-4c81d139eb8a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:28.121048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1b279a35-6d17-4e86-95c6-4c81d139eb8a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:28.132477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b279a35-6d17-4e86-95c6-4c81d139eb8a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:28.134109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b279a35-6d17-4e86-95c6-4c81d139eb8a service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:28.143886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1168/2317] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:33.285244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:33.299256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3bdfa9-f948-48e7-beb5-22c5b9b637a0 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:33.305361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:33.306247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:33.306526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:33.306953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:33.309539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:33.323669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:33.325813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:33.338816 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1150/2318] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:36.690244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f3d3c9-f949-47b8-9e23-007ed56a0121 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:36.690775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f3d3c9-f949-47b8-9e23-007ed56a0121 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:36.691127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f3d3c9-f949-47b8-9e23-007ed56a0121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:36.710757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d9f3d3c9-f949-47b8-9e23-007ed56a0121 None None] MFA Rules not processed for user `3326fcf3ccbc4a978e9e52e25456ad98`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:19:36.711990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d9f3d3c9-f949-47b8-9e23-007ed56a0121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:36.719022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1169/2319] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:19:36 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:19:36.927545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:36.929081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:36.987989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea53e462-a31b-4009-8b6c-9124967dbe05 service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:36.991643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:36.992280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:36.992588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:36.992861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:36.994612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:37.006119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:37.007550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:37.016553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1151/2320] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:36 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:37.041145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f3d3c9-f949-47b8-9e23-007ed56a0121 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:37.050648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f3d3c9-f949-47b8-9e23-007ed56a0121 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:37.054942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:37.055544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:37.055823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:37.056094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:37.057898 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:37.071865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:37.073519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:37.082172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1170/2321] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:37.999119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:19:38.014366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9137a1-2cfc-4666-875e-ba0e325bddf6 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:19:38.020508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:19:38.021112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:19:38.021486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:19:38.021766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:19:38.026488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:19:38.035523 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:19:38.037096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:19:38.047999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1152/2322] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:19:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:19:47.077968 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=44 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:19:48.043256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=72 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:02.591862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:02.603481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709e285d-75f5-495b-8c07-c44c71c3550e service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:02.607042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:02.607632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:02.607911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:02.608183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:02.609879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:02.722742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:02.724156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:02.743717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1171/2323] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:20:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:20:06.339989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:06.341573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:06.389832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee922509-d182-490f-9de2-32791dcc072a service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:06.395085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:06.395752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:06.395978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:06.396246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] PATH_INFO: `/v3/users/b6ca9962541744bf9d67cc3e05721413` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:06.397364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6ca9962541744bf9d67cc3e05721413)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:06.399307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:06.602591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6ca9962541744bf9d67cc3e05721413'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:06.617469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6ca9962541744bf9d67cc3e05721413'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:06.617469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6ca9962541744bf9d67cc3e05721413'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:06.713878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6ca9962541744bf9d67cc3e05721413'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:06.721859 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-864070eb-e7aa-460d-b9f0-7f0c7d0ade0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6ca9962541744bf9d67cc3e05721413'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:06.727629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1153/2324] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:20:06 2022] DELETE /identity/v3/users/b6ca9962541744bf9d67cc3e05721413 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:06.983879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:06.994294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78acf456-f520-4f9f-ba17-20002147ff20 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:07.052585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:07.053211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.053490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.053792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] PATH_INFO: `/v3/projects/8afac3f36b8449aa8a25a376aa2c3ed9` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.054899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] RBAC: Authorizing `identity:delete_project(project_id=8afac3f36b8449aa8a25a376aa2c3ed9)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:07.056870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:07.185208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:07.194836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8afac3f36b8449aa8a25a376aa2c3ed9', 'request_id': 'req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:07.200205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8afac3f36b8449aa8a25a376aa2c3ed9', 'request_id': 'req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:07.274042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8afac3f36b8449aa8a25a376aa2c3ed9', 'request_id': 'req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:07.286849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] The token cache is being invalidate because project 8afac3f36b8449aa8a25a376aa2c3ed9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:20:07.287133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:07.287843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-e1ee8359-dc29-4b4e-ac1e-0a839f69334e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:07.289317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1172/2325] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:20:06 2022] DELETE /identity/v3/projects/8afac3f36b8449aa8a25a376aa2c3ed9 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:07.294342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f017499-97dc-454f-baea-79da399c6324 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.294657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f017499-97dc-454f-baea-79da399c6324 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.294934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f017499-97dc-454f-baea-79da399c6324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.322699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3f017499-97dc-454f-baea-79da399c6324 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:07.323956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3f017499-97dc-454f-baea-79da399c6324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:07.381885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1154/2326] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:07.389486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c0fd8-0220-4347-b82a-6d4e07914f58 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.389823 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c0fd8-0220-4347-b82a-6d4e07914f58 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.390098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c0fd8-0220-4347-b82a-6d4e07914f58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.410179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-098c0fd8-0220-4347-b82a-6d4e07914f58 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:07.411375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-098c0fd8-0220-4347-b82a-6d4e07914f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:07.419160 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1173/2327] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:07.423205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f017499-97dc-454f-baea-79da399c6324 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:07.432736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f017499-97dc-454f-baea-79da399c6324 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:07.437151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:07.437773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.438053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.438325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.438959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:20:07.439261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:07.440853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:07.452017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1155/2328] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:20:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:20:07.456560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098c0fd8-0220-4347-b82a-6d4e07914f58 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:07.465644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098c0fd8-0220-4347-b82a-6d4e07914f58 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:07.470204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8401d2-685b-4689-b2fc-baf9f4652853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:07.471689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8401d2-685b-4689-b2fc-baf9f4652853 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.471975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8401d2-685b-4689-b2fc-baf9f4652853 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.472244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8401d2-685b-4689-b2fc-baf9f4652853 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.472855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8401d2-685b-4689-b2fc-baf9f4652853 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:07.474768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8401d2-685b-4689-b2fc-baf9f4652853 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:07.606623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1174/2329] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:20:07 2022] POST /identity/v3/projects => generated 363 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:07.610712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:07.620888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10cecee-3607-451d-83e1-83ab125c7174 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:07.637858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:07.677046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66627126-69c6-4e2a-a264-12c8db87a1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:07.677683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66627126-69c6-4e2a-a264-12c8db87a1fa admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.678035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66627126-69c6-4e2a-a264-12c8db87a1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.678303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66627126-69c6-4e2a-a264-12c8db87a1fa admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.678956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66627126-69c6-4e2a-a264-12c8db87a1fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:07.680915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66627126-69c6-4e2a-a264-12c8db87a1fa admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:07.788812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1156/2330] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:20:07 2022] POST /identity/v3/users => generated 282 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:07.792895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8401d2-685b-4689-b2fc-baf9f4652853 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:07.803138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8401d2-685b-4689-b2fc-baf9f4652853 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:07.807880 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d0070-fb9b-426b-a2c6-204640f90a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:07.808463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d0070-fb9b-426b-a2c6-204640f90a62 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.808743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d0070-fb9b-426b-a2c6-204640f90a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.809010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d0070-fb9b-426b-a2c6-204640f90a62 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.809662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216d0070-fb9b-426b-a2c6-204640f90a62 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:07.811324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216d0070-fb9b-426b-a2c6-204640f90a62 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:07.821893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1175/2331] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:07.826086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66627126-69c6-4e2a-a264-12c8db87a1fa admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:07.836814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66627126-69c6-4e2a-a264-12c8db87a1fa admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:07.841467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c1e697-607e-4b0f-952d-aac11a202dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:07.842217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c1e697-607e-4b0f-952d-aac11a202dc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.842430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c1e697-607e-4b0f-952d-aac11a202dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.842706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c1e697-607e-4b0f-952d-aac11a202dc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.843338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c1e697-607e-4b0f-952d-aac11a202dc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:07.845073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c1e697-607e-4b0f-952d-aac11a202dc0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:07.856596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1157/2332] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:07.860747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d0070-fb9b-426b-a2c6-204640f90a62 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:07.870509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d0070-fb9b-426b-a2c6-204640f90a62 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:07.875141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d477b-f498-4c28-be97-338b03a377f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:07.875789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d477b-f498-4c28-be97-338b03a377f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.876071 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d477b-f498-4c28-be97-338b03a377f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.876348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84d477b-f498-4c28-be97-338b03a377f7 admin admin] PATH_INFO: `/v3/projects/f1a56cfcac844af9af2d5754ca65b104/users/d7f8357cc41e4e3fb00612745591700f/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:07.891450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84d477b-f498-4c28-be97-338b03a377f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1a56cfcac844af9af2d5754ca65b104, user_id=d7f8357cc41e4e3fb00612745591700f, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:07.894782 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84d477b-f498-4c28-be97-338b03a377f7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:07.965306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1176/2333] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:07 2022] PUT /identity/v3/projects/f1a56cfcac844af9af2d5754ca65b104/users/d7f8357cc41e4e3fb00612745591700f/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:07.976974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93749da9-7b70-4d3b-8d34-09ea44337525 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:07.977259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93749da9-7b70-4d3b-8d34-09ea44337525 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:07.977531 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93749da9-7b70-4d3b-8d34-09ea44337525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.004559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-93749da9-7b70-4d3b-8d34-09ea44337525 None None] MFA Rules not processed for user `d7f8357cc41e4e3fb00612745591700f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:08.005806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-93749da9-7b70-4d3b-8d34-09ea44337525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:08.072179 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1158/2334] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:07 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:08.076672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d477b-f498-4c28-be97-338b03a377f7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:08.138815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84d477b-f498-4c28-be97-338b03a377f7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:08.143547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b575442-dd0a-4bec-9559-809f170fcbdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:08.144145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b575442-dd0a-4bec-9559-809f170fcbdd admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:08.144427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b575442-dd0a-4bec-9559-809f170fcbdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:08.144701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b575442-dd0a-4bec-9559-809f170fcbdd admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.145326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b575442-dd0a-4bec-9559-809f170fcbdd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:08.147379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b575442-dd0a-4bec-9559-809f170fcbdd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:08.253128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1177/2335] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:20:08 2022] POST /identity/v3/projects => generated 365 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:08.257193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93749da9-7b70-4d3b-8d34-09ea44337525 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:08.268661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93749da9-7b70-4d3b-8d34-09ea44337525 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:08.322994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0d10fd-ba8a-4d2e-968e-fe2d91e9b940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:08.323595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0d10fd-ba8a-4d2e-968e-fe2d91e9b940 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:08.323870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0d10fd-ba8a-4d2e-968e-fe2d91e9b940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:08.324139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0d10fd-ba8a-4d2e-968e-fe2d91e9b940 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.324856 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0d10fd-ba8a-4d2e-968e-fe2d91e9b940 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:08.326787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0d10fd-ba8a-4d2e-968e-fe2d91e9b940 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:08.411839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1159/2336] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:20:08 2022] POST /identity/v3/users => generated 289 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:08.415795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b575442-dd0a-4bec-9559-809f170fcbdd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:08.426583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b575442-dd0a-4bec-9559-809f170fcbdd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:08.431251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c148a3-2073-46d2-86d1-ab2d6e8893a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:08.431848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c148a3-2073-46d2-86d1-ab2d6e8893a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:08.432125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c148a3-2073-46d2-86d1-ab2d6e8893a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:08.432399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c148a3-2073-46d2-86d1-ab2d6e8893a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.433039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c148a3-2073-46d2-86d1-ab2d6e8893a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:08.434689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c148a3-2073-46d2-86d1-ab2d6e8893a0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:08.445279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1178/2337] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:08 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:08.449272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0d10fd-ba8a-4d2e-968e-fe2d91e9b940 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:08.459884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0d10fd-ba8a-4d2e-968e-fe2d91e9b940 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:08.464492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c401575-fa97-49db-b1a5-7e2264a84e52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:08.465127 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c401575-fa97-49db-b1a5-7e2264a84e52 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:08.465404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c401575-fa97-49db-b1a5-7e2264a84e52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:08.465710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c401575-fa97-49db-b1a5-7e2264a84e52 admin admin] PATH_INFO: `/v3/projects/24d0eeebc5ee4e18b2a854db58412756/users/dba6520c7a4b4dcdb7c84afd5bb7fab0/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.480916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c401575-fa97-49db-b1a5-7e2264a84e52 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24d0eeebc5ee4e18b2a854db58412756, user_id=dba6520c7a4b4dcdb7c84afd5bb7fab0, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:08.483620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c401575-fa97-49db-b1a5-7e2264a84e52 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:08.485776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:08.582847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1160/2338] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:08 2022] PUT /identity/v3/projects/24d0eeebc5ee4e18b2a854db58412756/users/dba6520c7a4b4dcdb7c84afd5bb7fab0/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:08.587121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c148a3-2073-46d2-86d1-ab2d6e8893a0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:08.597649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c148a3-2073-46d2-86d1-ab2d6e8893a0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:08.626749 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:08.651914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd0fe7a-cb30-4196-818b-b8820390a982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:08.652527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0fe7a-cb30-4196-818b-b8820390a982 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:08.652808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0fe7a-cb30-4196-818b-b8820390a982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:08.653080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0fe7a-cb30-4196-818b-b8820390a982 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.653747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd0fe7a-cb30-4196-818b-b8820390a982 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:08.655375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd0fe7a-cb30-4196-818b-b8820390a982 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:08.666711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1179/2339] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:08 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:20:08.670255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c401575-fa97-49db-b1a5-7e2264a84e52 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:08.681134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c401575-fa97-49db-b1a5-7e2264a84e52 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:08.685815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:08.686465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:08.686737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:08.687010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] PATH_INFO: `/v3/projects/24d0eeebc5ee4e18b2a854db58412756/users/dba6520c7a4b4dcdb7c84afd5bb7fab0/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.688912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24d0eeebc5ee4e18b2a854db58412756, user_id=dba6520c7a4b4dcdb7c84afd5bb7fab0, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:08.691607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:08.701718 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1161/2340] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:08 2022] PUT /identity/v3/projects/24d0eeebc5ee4e18b2a854db58412756/users/dba6520c7a4b4dcdb7c84afd5bb7fab0/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:20:08.711937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff83aea-5c65-416d-8a91-c7e3508b5b14 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:08.712253 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff83aea-5c65-416d-8a91-c7e3508b5b14 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:08.712525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff83aea-5c65-416d-8a91-c7e3508b5b14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.740074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-dff83aea-5c65-416d-8a91-c7e3508b5b14 None None] MFA Rules not processed for user `dba6520c7a4b4dcdb7c84afd5bb7fab0`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:08.741261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-dff83aea-5c65-416d-8a91-c7e3508b5b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:08.808345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1180/2341] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:08 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:20:08.820100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:08.821810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:08.878248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0691179-4058-46e4-97f5-d76378b81b14 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:08.882383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff15ad15-bb41-423f-84bc-2d24064cb2b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:08.882977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff15ad15-bb41-423f-84bc-2d24064cb2b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:08.883256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff15ad15-bb41-423f-84bc-2d24064cb2b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:08.883527 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff15ad15-bb41-423f-84bc-2d24064cb2b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:08.892320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff15ad15-bb41-423f-84bc-2d24064cb2b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:08.893781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff15ad15-bb41-423f-84bc-2d24064cb2b4 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:08.903057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1162/2342] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:20:08 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:20:09.809437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff83aea-5c65-416d-8a91-c7e3508b5b14 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:09.867141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff83aea-5c65-416d-8a91-c7e3508b5b14 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:09.871579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:09.872193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:09.872468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:09.872737 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] PATH_INFO: `/v3/users/d7f8357cc41e4e3fb00612745591700f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:09.873860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7f8357cc41e4e3fb00612745591700f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:09.875742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:10.121571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7f8357cc41e4e3fb00612745591700f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:10.136108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7f8357cc41e4e3fb00612745591700f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:10.136108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:10.136108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7f8357cc41e4e3fb00612745591700f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:10.250931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7f8357cc41e4e3fb00612745591700f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:10.258396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7f8357cc41e4e3fb00612745591700f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:10.264321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1181/2343] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:20:09 2022] DELETE /identity/v3/users/d7f8357cc41e4e3fb00612745591700f => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:10.268924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0350df-3d4a-4fed-85ec-8c66990d275b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:10.269361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0350df-3d4a-4fed-85ec-8c66990d275b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:10.269476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0350df-3d4a-4fed-85ec-8c66990d275b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:10.290147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-9f0350df-3d4a-4fed-85ec-8c66990d275b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:10.291344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9f0350df-3d4a-4fed-85ec-8c66990d275b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:10.350189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1163/2344] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:10.360143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:10.361557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:10.419800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89064825-0ea8-4b85-9722-738235ee07fd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:10.421955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477914f8-529c-47a3-be13-63e46d0920ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:10.422567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477914f8-529c-47a3-be13-63e46d0920ae service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:10.422840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477914f8-529c-47a3-be13-63e46d0920ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:10.423106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477914f8-529c-47a3-be13-63e46d0920ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:10.431842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477914f8-529c-47a3-be13-63e46d0920ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:10.433255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477914f8-529c-47a3-be13-63e46d0920ae service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:10.442764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1182/2345] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:20:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:20:10.886801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0350df-3d4a-4fed-85ec-8c66990d275b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:10.888379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9f0350df-3d4a-4fed-85ec-8c66990d275b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:10.900239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0350df-3d4a-4fed-85ec-8c66990d275b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:10.904965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:10.905618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:10.905900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:10.906172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] PATH_INFO: `/v3/projects/f1a56cfcac844af9af2d5754ca65b104` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:10.907346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1a56cfcac844af9af2d5754ca65b104)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:10.909251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:11.016877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1a56cfcac844af9af2d5754ca65b104', 'request_id': 'req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.120016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1a56cfcac844af9af2d5754ca65b104', 'request_id': 'req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.126275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1a56cfcac844af9af2d5754ca65b104', 'request_id': 'req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.137651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] The token cache is being invalidate because project f1a56cfcac844af9af2d5754ca65b104 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:20:11.137940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.138675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.140172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1164/2346] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:20:10 2022] DELETE /identity/v3/projects/f1a56cfcac844af9af2d5754ca65b104 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:11.144121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477914f8-529c-47a3-be13-63e46d0920ae service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:11.154181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477914f8-529c-47a3-be13-63e46d0920ae service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:11.209675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:11.210296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:11.210571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:11.210841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] PATH_INFO: `/v3/users/dba6520c7a4b4dcdb7c84afd5bb7fab0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:11.211941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dba6520c7a4b4dcdb7c84afd5bb7fab0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:11.213860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:11.412873 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dba6520c7a4b4dcdb7c84afd5bb7fab0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.413515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dba6520c7a4b4dcdb7c84afd5bb7fab0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.425269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dba6520c7a4b4dcdb7c84afd5bb7fab0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.525203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dba6520c7a4b4dcdb7c84afd5bb7fab0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.530606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:11.532511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-bbb6d279-bec0-4c66-84d8-5e5d23890358 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dba6520c7a4b4dcdb7c84afd5bb7fab0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:11.538418 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1183/2347] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:20:11 2022] DELETE /identity/v3/users/dba6520c7a4b4dcdb7c84afd5bb7fab0 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:12.058914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:12.060388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:12.076438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac4d3a9-c618-49a0-92d7-cbbe519f50d7 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:12.080138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:INSERT=3,keystone:SELECT=90,keystone:DELETE=7 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:12.135392 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:12.136025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.136301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.136572 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] PATH_INFO: `/v3/projects/24d0eeebc5ee4e18b2a854db58412756` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:12.137785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] RBAC: Authorizing `identity:delete_project(project_id=24d0eeebc5ee4e18b2a854db58412756)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:12.139725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:12.257048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24d0eeebc5ee4e18b2a854db58412756', 'request_id': 'req-70032c37-b434-406c-977a-182e7ffb4694'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:12.330794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24d0eeebc5ee4e18b2a854db58412756', 'request_id': 'req-70032c37-b434-406c-977a-182e7ffb4694'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:12.336997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24d0eeebc5ee4e18b2a854db58412756', 'request_id': 'req-70032c37-b434-406c-977a-182e7ffb4694'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:12.348976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] The token cache is being invalidate because project 24d0eeebc5ee4e18b2a854db58412756 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:20:12.349274 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:12.349914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-70032c37-b434-406c-977a-182e7ffb4694 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:12.351387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1165/2348] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:20:12 2022] DELETE /identity/v3/projects/24d0eeebc5ee4e18b2a854db58412756 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:12.356623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fa79eb-e7a2-4ea7-a5a9-08239c4ab1d8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.357112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fa79eb-e7a2-4ea7-a5a9-08239c4ab1d8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.357221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fa79eb-e7a2-4ea7-a5a9-08239c4ab1d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:12.376268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-18fa79eb-e7a2-4ea7-a5a9-08239c4ab1d8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:12.377451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-18fa79eb-e7a2-4ea7-a5a9-08239c4ab1d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:12.437120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1184/2349] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:12.445329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2d62a7-3f8f-4d68-bc98-06b0f0e2598c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.445667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2d62a7-3f8f-4d68-bc98-06b0f0e2598c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.445943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2d62a7-3f8f-4d68-bc98-06b0f0e2598c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:12.467142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6b2d62a7-3f8f-4d68-bc98-06b0f0e2598c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:12.468372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6b2d62a7-3f8f-4d68-bc98-06b0f0e2598c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:12.476681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1166/2350] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:12.480866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fa79eb-e7a2-4ea7-a5a9-08239c4ab1d8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:12.490466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fa79eb-e7a2-4ea7-a5a9-08239c4ab1d8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:12.495005 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:12.495595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.495873 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.496143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:12.496782 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:20:12.497085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:12.498697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:12.510472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1185/2351] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:20:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:20:12.514682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2d62a7-3f8f-4d68-bc98-06b0f0e2598c None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:12.524420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2d62a7-3f8f-4d68-bc98-06b0f0e2598c None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:12.529013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eeab0d-3722-4410-b3f5-bbd0658e2f3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:12.529651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eeab0d-3722-4410-b3f5-bbd0658e2f3c admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.529930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eeab0d-3722-4410-b3f5-bbd0658e2f3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.530198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eeab0d-3722-4410-b3f5-bbd0658e2f3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:12.530806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02eeab0d-3722-4410-b3f5-bbd0658e2f3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:12.532717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02eeab0d-3722-4410-b3f5-bbd0658e2f3c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:12.606505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1167/2352] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:20:12 2022] POST /identity/v3/projects => generated 365 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:20:12.610505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:12.620709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c620ab1-f67c-467c-84fa-4e497c45fd58 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:12.674218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77fce4-7472-4d57-acdf-88aa74cc056b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:12.674811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77fce4-7472-4d57-acdf-88aa74cc056b admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.675089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77fce4-7472-4d57-acdf-88aa74cc056b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.675358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77fce4-7472-4d57-acdf-88aa74cc056b admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:12.675977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77fce4-7472-4d57-acdf-88aa74cc056b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:12.678461 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77fce4-7472-4d57-acdf-88aa74cc056b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:12.793574 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1186/2353] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:20:12 2022] POST /identity/v3/users => generated 283 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:12.797644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eeab0d-3722-4410-b3f5-bbd0658e2f3c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:12.808541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eeab0d-3722-4410-b3f5-bbd0658e2f3c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:12.813464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3442c1-47b3-44d2-98ae-41dcc5626eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:12.814091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3442c1-47b3-44d2-98ae-41dcc5626eb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.814373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3442c1-47b3-44d2-98ae-41dcc5626eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.814646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3442c1-47b3-44d2-98ae-41dcc5626eb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:12.815305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3442c1-47b3-44d2-98ae-41dcc5626eb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:12.816895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c3442c1-47b3-44d2-98ae-41dcc5626eb2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:12.827426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1168/2354] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:12.831158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77fce4-7472-4d57-acdf-88aa74cc056b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:12.840801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77fce4-7472-4d57-acdf-88aa74cc056b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:12.845287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215c4c40-391f-4f3f-883f-426f86f80e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:12.846164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215c4c40-391f-4f3f-883f-426f86f80e7e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.846439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215c4c40-391f-4f3f-883f-426f86f80e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.846706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215c4c40-391f-4f3f-883f-426f86f80e7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:12.847336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215c4c40-391f-4f3f-883f-426f86f80e7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:12.848902 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215c4c40-391f-4f3f-883f-426f86f80e7e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:12.859208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1187/2355] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:12 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:12.863003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3442c1-47b3-44d2-98ae-41dcc5626eb2 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:12.873090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3442c1-47b3-44d2-98ae-41dcc5626eb2 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:12.877538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63db5ef3-5138-471b-8619-f06514140554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:12.878208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63db5ef3-5138-471b-8619-f06514140554 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:12.878490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63db5ef3-5138-471b-8619-f06514140554 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:12.878760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63db5ef3-5138-471b-8619-f06514140554 admin admin] PATH_INFO: `/v3/projects/0a4f4f2ed19a45bea762017113e3286e/users/4535944e87a34ac19d92c63063fc4e52/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.015341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63db5ef3-5138-471b-8619-f06514140554 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a4f4f2ed19a45bea762017113e3286e, user_id=4535944e87a34ac19d92c63063fc4e52, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:13.017984 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63db5ef3-5138-471b-8619-f06514140554 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:13.126004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1169/2356] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:12 2022] PUT /identity/v3/projects/0a4f4f2ed19a45bea762017113e3286e/users/4535944e87a34ac19d92c63063fc4e52/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:13.137574 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f795be-d289-427a-8ee0-ef67229aa315 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:13.137896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f795be-d289-427a-8ee0-ef67229aa315 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:13.138175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f795be-d289-427a-8ee0-ef67229aa315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.165826 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-73f795be-d289-427a-8ee0-ef67229aa315 None None] MFA Rules not processed for user `4535944e87a34ac19d92c63063fc4e52`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:13.167027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-73f795be-d289-427a-8ee0-ef67229aa315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:13.213276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:13.231899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1188/2357] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:13 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:13.236460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63db5ef3-5138-471b-8619-f06514140554 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:13.269885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:13.297163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63db5ef3-5138-471b-8619-f06514140554 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:13.301787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b482a480-c0b5-4931-943a-eb701d71d982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:13.302381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b482a480-c0b5-4931-943a-eb701d71d982 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:13.302654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b482a480-c0b5-4931-943a-eb701d71d982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:13.302921 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b482a480-c0b5-4931-943a-eb701d71d982 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.303564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b482a480-c0b5-4931-943a-eb701d71d982 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:13.305439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b482a480-c0b5-4931-943a-eb701d71d982 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:13.423847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1170/2358] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:20:13 2022] POST /identity/v3/projects => generated 365 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:13.427948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f795be-d289-427a-8ee0-ef67229aa315 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:13.438767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f795be-d289-427a-8ee0-ef67229aa315 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:13.494846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba632505-1ffe-4b7e-a172-9e8b8a329972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:13.495488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba632505-1ffe-4b7e-a172-9e8b8a329972 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:13.495763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba632505-1ffe-4b7e-a172-9e8b8a329972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:13.496026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba632505-1ffe-4b7e-a172-9e8b8a329972 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.496633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba632505-1ffe-4b7e-a172-9e8b8a329972 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:13.498595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba632505-1ffe-4b7e-a172-9e8b8a329972 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:13.622450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1189/2359] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:20:13 2022] POST /identity/v3/users => generated 289 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:13.626584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b482a480-c0b5-4931-943a-eb701d71d982 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:13.636570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b482a480-c0b5-4931-943a-eb701d71d982 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:13.641091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c439a02b-25f9-4dc5-acc9-34ebaa201e11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:13.641711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c439a02b-25f9-4dc5-acc9-34ebaa201e11 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:13.641989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c439a02b-25f9-4dc5-acc9-34ebaa201e11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:13.642258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c439a02b-25f9-4dc5-acc9-34ebaa201e11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.642889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c439a02b-25f9-4dc5-acc9-34ebaa201e11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:13.644474 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c439a02b-25f9-4dc5-acc9-34ebaa201e11 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:13.655029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1171/2360] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:13.664057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba632505-1ffe-4b7e-a172-9e8b8a329972 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:13.674065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba632505-1ffe-4b7e-a172-9e8b8a329972 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:13.678583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69c3465-f505-4a36-aa20-60c887494b7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:13.679217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c3465-f505-4a36-aa20-60c887494b7b admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:13.679497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c3465-f505-4a36-aa20-60c887494b7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:13.679769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69c3465-f505-4a36-aa20-60c887494b7b admin admin] PATH_INFO: `/v3/projects/f93f02069d924c01b0065bc82fec3492/users/489b48304e6f403ca64c5d8254a308b2/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.694923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69c3465-f505-4a36-aa20-60c887494b7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f93f02069d924c01b0065bc82fec3492, user_id=489b48304e6f403ca64c5d8254a308b2, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:13.697679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69c3465-f505-4a36-aa20-60c887494b7b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:13.805731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1190/2361] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:13 2022] PUT /identity/v3/projects/f93f02069d924c01b0065bc82fec3492/users/489b48304e6f403ca64c5d8254a308b2/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:13.809700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c439a02b-25f9-4dc5-acc9-34ebaa201e11 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:13.819498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c439a02b-25f9-4dc5-acc9-34ebaa201e11 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:13.874042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7fa6be-1212-49b4-86f3-c0d4218f09cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:13.874640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fa6be-1212-49b4-86f3-c0d4218f09cb admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:13.874922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fa6be-1212-49b4-86f3-c0d4218f09cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:13.875191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7fa6be-1212-49b4-86f3-c0d4218f09cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.875813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7fa6be-1212-49b4-86f3-c0d4218f09cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:13.877445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7fa6be-1212-49b4-86f3-c0d4218f09cb admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:13.886782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1172/2362] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:13 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:13.890831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69c3465-f505-4a36-aa20-60c887494b7b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:13.900956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69c3465-f505-4a36-aa20-60c887494b7b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:13.905627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:13.906261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:13.906537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:13.906808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] PATH_INFO: `/v3/projects/f93f02069d924c01b0065bc82fec3492/users/489b48304e6f403ca64c5d8254a308b2/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.908739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f93f02069d924c01b0065bc82fec3492, user_id=489b48304e6f403ca64c5d8254a308b2, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:13.911872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:13.921874 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1191/2363] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:13 2022] PUT /identity/v3/projects/f93f02069d924c01b0065bc82fec3492/users/489b48304e6f403ca64c5d8254a308b2/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:13.932810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1fced-8682-4dbe-b69a-c4d8cf24715e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:13.933036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1fced-8682-4dbe-b69a-c4d8cf24715e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:13.933311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1fced-8682-4dbe-b69a-c4d8cf24715e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:13.959727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ccd1fced-8682-4dbe-b69a-c4d8cf24715e None None] MFA Rules not processed for user `489b48304e6f403ca64c5d8254a308b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:13.960924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ccd1fced-8682-4dbe-b69a-c4d8cf24715e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:14.027703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1173/2364] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:13 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:14.039966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:14.041672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:14.099415 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363a52b-70d6-4736-ba83-12b8d817447e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:14.102996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4fdee-d2a7-47ac-a523-3cf76342e7ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:14.103591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4fdee-d2a7-47ac-a523-3cf76342e7ed service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:14.103866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4fdee-d2a7-47ac-a523-3cf76342e7ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:14.104135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4fdee-d2a7-47ac-a523-3cf76342e7ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:14.112919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e4fdee-d2a7-47ac-a523-3cf76342e7ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:14.114380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e4fdee-d2a7-47ac-a523-3cf76342e7ed service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:14.123559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1192/2365] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:20:14 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:20:14.356547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd1fced-8682-4dbe-b69a-c4d8cf24715e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:14.366304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd1fced-8682-4dbe-b69a-c4d8cf24715e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:14.369819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbb80f2-1323-4dd2-b580-f3a3ed738d90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:14.370406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbb80f2-1323-4dd2-b580-f3a3ed738d90 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:14.370678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbb80f2-1323-4dd2-b580-f3a3ed738d90 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:14.370947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbb80f2-1323-4dd2-b580-f3a3ed738d90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:14.401506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:14.439956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbb80f2-1323-4dd2-b580-f3a3ed738d90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:14.441397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbb80f2-1323-4dd2-b580-f3a3ed738d90 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:14.450823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1174/2366] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:20:14 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:20:24.119410 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=74,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:24.599237 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=15 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:28.362742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4fdee-d2a7-47ac-a523-3cf76342e7ed service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:28.425966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4fdee-d2a7-47ac-a523-3cf76342e7ed service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:28.431582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:28.432259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:28.432551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:28.432822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] PATH_INFO: `/v3/users/4535944e87a34ac19d92c63063fc4e52` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:28.433982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=4535944e87a34ac19d92c63063fc4e52)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:28.435887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:28.636889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4535944e87a34ac19d92c63063fc4e52'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:28.637547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4535944e87a34ac19d92c63063fc4e52'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:28.651300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4535944e87a34ac19d92c63063fc4e52'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:28.765199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4535944e87a34ac19d92c63063fc4e52'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:28.772796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4535944e87a34ac19d92c63063fc4e52'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:28.778758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1193/2367] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:20:28 2022] DELETE /identity/v3/users/4535944e87a34ac19d92c63063fc4e52 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:28.783221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c4643-db03-46de-9599-181fe7bb3fbc None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:28.783724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c4643-db03-46de-9599-181fe7bb3fbc None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:28.783862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c4643-db03-46de-9599-181fe7bb3fbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:28.802428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-292c4643-db03-46de-9599-181fe7bb3fbc None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:28.804158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-292c4643-db03-46de-9599-181fe7bb3fbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:28.862988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1175/2368] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:28.873065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:28.874494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:28.932779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2eaf49-61ea-468a-a693-ad706596fb0d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:28.936477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47932626-91d5-4731-8e87-c8b94c27956d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:28.941164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47932626-91d5-4731-8e87-c8b94c27956d service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:28.941164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47932626-91d5-4731-8e87-c8b94c27956d service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:28.941164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47932626-91d5-4731-8e87-c8b94c27956d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:28.947439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47932626-91d5-4731-8e87-c8b94c27956d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:28.948876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47932626-91d5-4731-8e87-c8b94c27956d service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:28.958973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1194/2369] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:20:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:20:29.488819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c4643-db03-46de-9599-181fe7bb3fbc None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:29.490315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-292c4643-db03-46de-9599-181fe7bb3fbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:29.502340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c4643-db03-46de-9599-181fe7bb3fbc None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:29.507129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:29.509395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:29.509395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:29.509395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] PATH_INFO: `/v3/projects/0a4f4f2ed19a45bea762017113e3286e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:29.510198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a4f4f2ed19a45bea762017113e3286e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:29.512124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:29.629560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a4f4f2ed19a45bea762017113e3286e', 'request_id': 'req-ee367139-6ec2-4f2b-99b2-ffa693bb657e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:29.708011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a4f4f2ed19a45bea762017113e3286e', 'request_id': 'req-ee367139-6ec2-4f2b-99b2-ffa693bb657e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:29.714439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a4f4f2ed19a45bea762017113e3286e', 'request_id': 'req-ee367139-6ec2-4f2b-99b2-ffa693bb657e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:29.727704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] The token cache is being invalidate because project 0a4f4f2ed19a45bea762017113e3286e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:20:29.728000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:29.728780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:29.731005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1176/2370] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:20:29 2022] DELETE /identity/v3/projects/0a4f4f2ed19a45bea762017113e3286e => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:29.735262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47932626-91d5-4731-8e87-c8b94c27956d service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:29.746003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47932626-91d5-4731-8e87-c8b94c27956d service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:29.765217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:29.805451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:29.806743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:29.807136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:29.807412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] PATH_INFO: `/v3/users/489b48304e6f403ca64c5d8254a308b2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:29.808618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] RBAC: Authorizing `identity:delete_user(user_id=489b48304e6f403ca64c5d8254a308b2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:29.810636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:30.039197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '489b48304e6f403ca64c5d8254a308b2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.040647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '489b48304e6f403ca64c5d8254a308b2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.052616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '489b48304e6f403ca64c5d8254a308b2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.113846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '489b48304e6f403ca64c5d8254a308b2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.122214 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-80e16d10-1064-4fed-833a-695d7b29932f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '489b48304e6f403ca64c5d8254a308b2'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.129490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1195/2371] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:20:29 2022] DELETE /identity/v3/users/489b48304e6f403ca64c5d8254a308b2 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:30.578772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:30.580203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:30.594039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee367139-6ec2-4f2b-99b2-ffa693bb657e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:30.626250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f664ea01-2591-4f8f-90f8-95b425def845 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:30.626569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f664ea01-2591-4f8f-90f8-95b425def845 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:30.626845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f664ea01-2591-4f8f-90f8-95b425def845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:30.646656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f664ea01-2591-4f8f-90f8-95b425def845 None None] MFA Rules not processed for user `d61cf1670a7a43519c7258cd467227b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:30.647899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f664ea01-2591-4f8f-90f8-95b425def845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:30.657971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:30.658673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:30.658957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:30.659227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] PATH_INFO: `/v3/projects/f93f02069d924c01b0065bc82fec3492` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:30.660363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f93f02069d924c01b0065bc82fec3492)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:30.662289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:30.703995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1196/2372] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:20:30.778159 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:30.794350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f93f02069d924c01b0065bc82fec3492', 'request_id': 'req-300e21d9-c8bf-4b87-9e93-08152b85d9d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.856375 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f93f02069d924c01b0065bc82fec3492', 'request_id': 'req-300e21d9-c8bf-4b87-9e93-08152b85d9d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.862340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f93f02069d924c01b0065bc82fec3492', 'request_id': 'req-300e21d9-c8bf-4b87-9e93-08152b85d9d7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.875090 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] The token cache is being invalidate because project f93f02069d924c01b0065bc82fec3492 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:20:30.875380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.876028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-300e21d9-c8bf-4b87-9e93-08152b85d9d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:20:30.877705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1177/2373] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:20:30 2022] DELETE /identity/v3/projects/f93f02069d924c01b0065bc82fec3492 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:20:30.882684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1a8fd-3a15-4ebf-be82-5b9cafd0beb7 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:30.883182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1a8fd-3a15-4ebf-be82-5b9cafd0beb7 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:30.883290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d1a8fd-3a15-4ebf-be82-5b9cafd0beb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:30.902926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-39d1a8fd-3a15-4ebf-be82-5b9cafd0beb7 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:30.904078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-39d1a8fd-3a15-4ebf-be82-5b9cafd0beb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:30.949405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:30.967269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1197/2374] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:30.977195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1439547e-1326-478b-9b53-d6dfd20af654 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:30.977544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1439547e-1326-478b-9b53-d6dfd20af654 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:30.977929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1439547e-1326-478b-9b53-d6dfd20af654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:30.998249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1439547e-1326-478b-9b53-d6dfd20af654 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:30.999468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1439547e-1326-478b-9b53-d6dfd20af654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:31.007209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1178/2375] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:20:31.011587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d1a8fd-3a15-4ebf-be82-5b9cafd0beb7 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:31.021109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d1a8fd-3a15-4ebf-be82-5b9cafd0beb7 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:31.026402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:31.026925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:31.027223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:31.027472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:31.028115 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:20:31.028413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:31.030047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:31.042047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1198/2376] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:20:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:20:31.046481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1439547e-1326-478b-9b53-d6dfd20af654 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:31.055803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1439547e-1326-478b-9b53-d6dfd20af654 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:31.060381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8c62c1-1661-4619-8322-d9e699b6f45c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:31.061120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c62c1-1661-4619-8322-d9e699b6f45c admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:31.061401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c62c1-1661-4619-8322-d9e699b6f45c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:31.061702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c62c1-1661-4619-8322-d9e699b6f45c admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:31.062320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8c62c1-1661-4619-8322-d9e699b6f45c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:31.064191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8c62c1-1661-4619-8322-d9e699b6f45c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:31.179007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1179/2377] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:20:31 2022] POST /identity/v3/projects => generated 377 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:31.183319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:31.194314 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63503514-91fa-4379-b4a7-a2af8c622c59 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:31.258602 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b65e062-7b49-4b28-9f7f-2a1d0229deaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:31.259230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b65e062-7b49-4b28-9f7f-2a1d0229deaf admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:31.259565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b65e062-7b49-4b28-9f7f-2a1d0229deaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:31.259866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b65e062-7b49-4b28-9f7f-2a1d0229deaf admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:31.260540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b65e062-7b49-4b28-9f7f-2a1d0229deaf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:31.262583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b65e062-7b49-4b28-9f7f-2a1d0229deaf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:31.455930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1199/2378] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:20:31 2022] POST /identity/v3/users => generated 289 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:20:31.456556 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8c62c1-1661-4619-8322-d9e699b6f45c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:31.467824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8c62c1-1661-4619-8322-d9e699b6f45c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:31.472624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f3a59-0578-4af8-924e-347ddf5d39d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:31.473219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f3a59-0578-4af8-924e-347ddf5d39d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:31.473495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f3a59-0578-4af8-924e-347ddf5d39d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:31.473794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50f3a59-0578-4af8-924e-347ddf5d39d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:31.474431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50f3a59-0578-4af8-924e-347ddf5d39d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:31.476012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50f3a59-0578-4af8-924e-347ddf5d39d6 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:31.486048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1180/2379] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:31.492709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b65e062-7b49-4b28-9f7f-2a1d0229deaf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:31.502988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b65e062-7b49-4b28-9f7f-2a1d0229deaf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:31.508159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae1d2d3-b37a-4c1e-9a69-5290ba405fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:31.508804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae1d2d3-b37a-4c1e-9a69-5290ba405fe9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:31.509082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae1d2d3-b37a-4c1e-9a69-5290ba405fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:31.509352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae1d2d3-b37a-4c1e-9a69-5290ba405fe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:31.510030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae1d2d3-b37a-4c1e-9a69-5290ba405fe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:31.511628 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae1d2d3-b37a-4c1e-9a69-5290ba405fe9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:31.529988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1200/2380] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:31.532991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f3a59-0578-4af8-924e-347ddf5d39d6 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:31.543070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50f3a59-0578-4af8-924e-347ddf5d39d6 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:31.547473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878ade15-120b-4524-9e87-33681a0c18fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:31.548115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878ade15-120b-4524-9e87-33681a0c18fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:31.548390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878ade15-120b-4524-9e87-33681a0c18fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:31.548663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878ade15-120b-4524-9e87-33681a0c18fd admin admin] PATH_INFO: `/v3/projects/9053d620fcca42a89e8b6028c30a3ff5/users/4415a418034345ed90e73367d5213e7c/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:31.563401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878ade15-120b-4524-9e87-33681a0c18fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=9053d620fcca42a89e8b6028c30a3ff5, user_id=4415a418034345ed90e73367d5213e7c, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:31.566083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878ade15-120b-4524-9e87-33681a0c18fd admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:31.675295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1181/2381] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:31 2022] PUT /identity/v3/projects/9053d620fcca42a89e8b6028c30a3ff5/users/4415a418034345ed90e73367d5213e7c/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:31.686440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89a6d-4fe4-4ed4-8a18-48201e801424 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:31.686727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89a6d-4fe4-4ed4-8a18-48201e801424 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:31.686998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c89a6d-4fe4-4ed4-8a18-48201e801424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:31.718595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-d3c89a6d-4fe4-4ed4-8a18-48201e801424 None None] MFA Rules not processed for user `4415a418034345ed90e73367d5213e7c`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:31.719861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d3c89a6d-4fe4-4ed4-8a18-48201e801424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:31.796145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1201/2382] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:31 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:31.800782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878ade15-120b-4524-9e87-33681a0c18fd admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:31.864394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878ade15-120b-4524-9e87-33681a0c18fd admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:31.869566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f04cf50-d68c-4d2c-809b-bca7860aba58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:31.870186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f04cf50-d68c-4d2c-809b-bca7860aba58 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:31.870460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f04cf50-d68c-4d2c-809b-bca7860aba58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:31.870728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f04cf50-d68c-4d2c-809b-bca7860aba58 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:31.871370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f04cf50-d68c-4d2c-809b-bca7860aba58 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:31.873244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f04cf50-d68c-4d2c-809b-bca7860aba58 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:31.943803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1182/2383] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:20:31 2022] POST /identity/v3/projects => generated 377 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:31.948565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c89a6d-4fe4-4ed4-8a18-48201e801424 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:31.959663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c89a6d-4fe4-4ed4-8a18-48201e801424 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:31.999404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:32.015788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377c45f6-3fbb-4f77-b713-b766725ef846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:32.016392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377c45f6-3fbb-4f77-b713-b766725ef846 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:32.016673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377c45f6-3fbb-4f77-b713-b766725ef846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:32.016941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377c45f6-3fbb-4f77-b713-b766725ef846 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:32.017551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377c45f6-3fbb-4f77-b713-b766725ef846 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:32.019475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377c45f6-3fbb-4f77-b713-b766725ef846 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:32.113778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1202/2384] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:20:31 2022] POST /identity/v3/users => generated 295 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:20:32.118702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f04cf50-d68c-4d2c-809b-bca7860aba58 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:32.129692 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f04cf50-d68c-4d2c-809b-bca7860aba58 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:32.134263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83438dc6-1873-42e3-a34c-c55a3c0aae24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:32.134851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83438dc6-1873-42e3-a34c-c55a3c0aae24 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:32.139639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83438dc6-1873-42e3-a34c-c55a3c0aae24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:32.139639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83438dc6-1873-42e3-a34c-c55a3c0aae24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:32.139639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83438dc6-1873-42e3-a34c-c55a3c0aae24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:32.139639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83438dc6-1873-42e3-a34c-c55a3c0aae24 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:32.147910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1183/2385] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:32.151988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377c45f6-3fbb-4f77-b713-b766725ef846 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:32.162158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377c45f6-3fbb-4f77-b713-b766725ef846 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:32.167769 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6037a-605d-420c-9f62-9900dd55f689 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:32.168470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6037a-605d-420c-9f62-9900dd55f689 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:32.168753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6037a-605d-420c-9f62-9900dd55f689 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:32.174055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef6037a-605d-420c-9f62-9900dd55f689 admin admin] PATH_INFO: `/v3/projects/3b661dfbc9d8428cb212d30377e226c8/users/41a5b91ab1304b0283eac74461807c8a/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:32.184454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef6037a-605d-420c-9f62-9900dd55f689 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b661dfbc9d8428cb212d30377e226c8, user_id=41a5b91ab1304b0283eac74461807c8a, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:32.187317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef6037a-605d-420c-9f62-9900dd55f689 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:32.274527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1203/2386] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:32 2022] PUT /identity/v3/projects/3b661dfbc9d8428cb212d30377e226c8/users/41a5b91ab1304b0283eac74461807c8a/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:32.278658 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83438dc6-1873-42e3-a34c-c55a3c0aae24 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:32.289552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83438dc6-1873-42e3-a34c-c55a3c0aae24 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:32.310037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:32.345073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b486e8b5-eb65-41f8-9057-da788ba56be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:32.346089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b486e8b5-eb65-41f8-9057-da788ba56be6 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:32.346396 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b486e8b5-eb65-41f8-9057-da788ba56be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:32.346776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b486e8b5-eb65-41f8-9057-da788ba56be6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:32.347407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b486e8b5-eb65-41f8-9057-da788ba56be6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:32.348990 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b486e8b5-eb65-41f8-9057-da788ba56be6 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:32.358929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1184/2387] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:20:32 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:20:32.363140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6037a-605d-420c-9f62-9900dd55f689 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:32.373993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef6037a-605d-420c-9f62-9900dd55f689 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:32.378936 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:32.379615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:32.379895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:32.380168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] PATH_INFO: `/v3/projects/3b661dfbc9d8428cb212d30377e226c8/users/41a5b91ab1304b0283eac74461807c8a/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:32.382156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b661dfbc9d8428cb212d30377e226c8, user_id=41a5b91ab1304b0283eac74461807c8a, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:32.384794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:32.395375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1204/2388] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:20:32 2022] PUT /identity/v3/projects/3b661dfbc9d8428cb212d30377e226c8/users/41a5b91ab1304b0283eac74461807c8a/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:20:32.406304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d7762-e180-4726-8af8-c93fe841e001 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:32.406592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d7762-e180-4726-8af8-c93fe841e001 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:32.406865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d7762-e180-4726-8af8-c93fe841e001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:32.434121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-242d7762-e180-4726-8af8-c93fe841e001 None None] MFA Rules not processed for user `41a5b91ab1304b0283eac74461807c8a`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:20:32.435303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-242d7762-e180-4726-8af8-c93fe841e001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:32.503025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1185/2389] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:20:32 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:20:32.514313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:32.515985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:32.575116 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44714a-bba8-4d62-b257-7cd00bf74ba7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:32.579951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f8500e-3d3f-4222-b29d-4eb39f0d57b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:32.580581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f8500e-3d3f-4222-b29d-4eb39f0d57b7 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:32.580856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f8500e-3d3f-4222-b29d-4eb39f0d57b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:32.581123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f8500e-3d3f-4222-b29d-4eb39f0d57b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:32.638564 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f8500e-3d3f-4222-b29d-4eb39f0d57b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:32.639973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f8500e-3d3f-4222-b29d-4eb39f0d57b7 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:32.648750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1205/2390] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:20:32 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:20:41.005336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242d7762-e180-4726-8af8-c93fe841e001 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:20:41.007365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-242d7762-e180-4726-8af8-c93fe841e001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:20:41.019264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242d7762-e180-4726-8af8-c93fe841e001 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:20:41.023440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f435d79-b4d7-4c33-a144-454507761ef6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:20:41.024043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f435d79-b4d7-4c33-a144-454507761ef6 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:20:41.024321 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f435d79-b4d7-4c33-a144-454507761ef6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:20:41.034338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f435d79-b4d7-4c33-a144-454507761ef6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:20:41.034338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f435d79-b4d7-4c33-a144-454507761ef6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:20:41.035353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f435d79-b4d7-4c33-a144-454507761ef6 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:20:41.045017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1186/2391] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:20:41 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:20:42.644616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=73,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:20:51.041100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=58 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:12.425112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f8500e-3d3f-4222-b29d-4eb39f0d57b7 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:12.487168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f8500e-3d3f-4222-b29d-4eb39f0d57b7 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:12.491702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:12.492327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:12.492611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:12.492885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] PATH_INFO: `/v3/users/4415a418034345ed90e73367d5213e7c` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:12.494030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4415a418034345ed90e73367d5213e7c)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:12.496662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:12.709582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4415a418034345ed90e73367d5213e7c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:12.710202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4415a418034345ed90e73367d5213e7c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:12.722759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4415a418034345ed90e73367d5213e7c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:12.798776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4415a418034345ed90e73367d5213e7c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:12.806079 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4415a418034345ed90e73367d5213e7c'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:12.811917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1206/2392] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:21:12 2022] DELETE /identity/v3/users/4415a418034345ed90e73367d5213e7c => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:12.816651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2581b8-412c-4da6-8ce0-ae3b4627ca58 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:12.817134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2581b8-412c-4da6-8ce0-ae3b4627ca58 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:12.817250 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2581b8-412c-4da6-8ce0-ae3b4627ca58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:12.835535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-0b2581b8-412c-4da6-8ce0-ae3b4627ca58 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:12.836700 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0b2581b8-412c-4da6-8ce0-ae3b4627ca58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:12.893254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1187/2393] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:21:12.905158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:12.906551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:12.963159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552c64d9-629d-4dcd-ace4-33ce5ea20293 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:12.967158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b739602-7b2e-49bc-85d7-6eab37deba0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:12.967754 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b739602-7b2e-49bc-85d7-6eab37deba0d service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:12.968032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b739602-7b2e-49bc-85d7-6eab37deba0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:12.968301 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b739602-7b2e-49bc-85d7-6eab37deba0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:12.977643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b739602-7b2e-49bc-85d7-6eab37deba0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:12.979056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b739602-7b2e-49bc-85d7-6eab37deba0d service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:12.988805 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1207/2394] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:21:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:21:13.436730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2581b8-412c-4da6-8ce0-ae3b4627ca58 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:13.438219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0b2581b8-412c-4da6-8ce0-ae3b4627ca58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:13.450582 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2581b8-412c-4da6-8ce0-ae3b4627ca58 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:13.455265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:13.455906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:13.456175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:13.456444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] PATH_INFO: `/v3/projects/9053d620fcca42a89e8b6028c30a3ff5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:13.457552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9053d620fcca42a89e8b6028c30a3ff5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:13.459487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:13.599436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9053d620fcca42a89e8b6028c30a3ff5', 'request_id': 'req-ea3c94da-1f7e-45b2-a226-b3468ae8168f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:13.707584 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9053d620fcca42a89e8b6028c30a3ff5', 'request_id': 'req-ea3c94da-1f7e-45b2-a226-b3468ae8168f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:13.713479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9053d620fcca42a89e8b6028c30a3ff5', 'request_id': 'req-ea3c94da-1f7e-45b2-a226-b3468ae8168f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:13.725011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] The token cache is being invalidate because project 9053d620fcca42a89e8b6028c30a3ff5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:21:13.725296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:13.725941 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:13.727409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1188/2395] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:21:13 2022] DELETE /identity/v3/projects/9053d620fcca42a89e8b6028c30a3ff5 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:13.731526 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b739602-7b2e-49bc-85d7-6eab37deba0d service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:13.740950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b739602-7b2e-49bc-85d7-6eab37deba0d service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:13.759088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:13.795354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:13.795972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:13.796248 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:13.796519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] PATH_INFO: `/v3/users/41a5b91ab1304b0283eac74461807c8a` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:13.797654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41a5b91ab1304b0283eac74461807c8a)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:13.799973 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:14.035292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41a5b91ab1304b0283eac74461807c8a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.035958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41a5b91ab1304b0283eac74461807c8a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.047595 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41a5b91ab1304b0283eac74461807c8a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.130246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41a5b91ab1304b0283eac74461807c8a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.137814 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-eb8fa91d-3e71-45ae-8336-6edda8b192c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41a5b91ab1304b0283eac74461807c8a'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.143613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1208/2396] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:21:13 2022] DELETE /identity/v3/users/41a5b91ab1304b0283eac74461807c8a => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:14.543669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:14.545254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:14.559640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c94da-1f7e-45b2-a226-b3468ae8168f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:14.616591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:14.617218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:14.617499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:14.617801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] PATH_INFO: `/v3/projects/3b661dfbc9d8428cb212d30377e226c8` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:14.618934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b661dfbc9d8428cb212d30377e226c8)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:14.621347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:14.754708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:14.770314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b661dfbc9d8428cb212d30377e226c8', 'request_id': 'req-7b9e27f5-b160-448a-a329-9c82d5b08963'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.878546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b661dfbc9d8428cb212d30377e226c8', 'request_id': 'req-7b9e27f5-b160-448a-a329-9c82d5b08963'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.884372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b661dfbc9d8428cb212d30377e226c8', 'request_id': 'req-7b9e27f5-b160-448a-a329-9c82d5b08963'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.895276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] The token cache is being invalidate because project 3b661dfbc9d8428cb212d30377e226c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:21:14.895599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.896218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7b9e27f5-b160-448a-a329-9c82d5b08963 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:14.897687 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1189/2397] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:21:14 2022] DELETE /identity/v3/projects/3b661dfbc9d8428cb212d30377e226c8 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:14.902518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb9492-d8b2-40c3-a252-4b3f9bfb786f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:14.902989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb9492-d8b2-40c3-a252-4b3f9bfb786f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:14.903091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb9492-d8b2-40c3-a252-4b3f9bfb786f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:14.923842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2bbb9492-d8b2-40c3-a252-4b3f9bfb786f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:14.925034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2bbb9492-d8b2-40c3-a252-4b3f9bfb786f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:14.983848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1209/2398] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:14.991862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31787101-bca0-4be7-9d40-e7c4bf2b2f64 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:14.992169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31787101-bca0-4be7-9d40-e7c4bf2b2f64 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:14.992436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31787101-bca0-4be7-9d40-e7c4bf2b2f64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.011963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-31787101-bca0-4be7-9d40-e7c4bf2b2f64 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:15.013071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-31787101-bca0-4be7-9d40-e7c4bf2b2f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:15.020752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1190/2399] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:15.024984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbb9492-d8b2-40c3-a252-4b3f9bfb786f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:15.034951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbb9492-d8b2-40c3-a252-4b3f9bfb786f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:15.039502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:15.040091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.040379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.040638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.041277 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:21:15.041579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:15.043760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:15.055053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1210/2400] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:21:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:21:15.059112 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31787101-bca0-4be7-9d40-e7c4bf2b2f64 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:15.068072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31787101-bca0-4be7-9d40-e7c4bf2b2f64 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:15.072730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c243e15-7288-4da8-bddc-0b84a4d041f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:15.073338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c243e15-7288-4da8-bddc-0b84a4d041f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.073705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c243e15-7288-4da8-bddc-0b84a4d041f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.073981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c243e15-7288-4da8-bddc-0b84a4d041f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.074595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c243e15-7288-4da8-bddc-0b84a4d041f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:15.076463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c243e15-7288-4da8-bddc-0b84a4d041f1 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:15.205949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1191/2401] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:21:15 2022] POST /identity/v3/projects => generated 383 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:21:15.210063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:15.220369 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c983480-7616-4a6f-b731-d9c6a171c2a9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:15.247252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:15.277967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb92dc48-e102-4bec-ba54-d01c5b77bd8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:15.278728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb92dc48-e102-4bec-ba54-d01c5b77bd8a admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.279195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb92dc48-e102-4bec-ba54-d01c5b77bd8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.279860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb92dc48-e102-4bec-ba54-d01c5b77bd8a admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.280494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb92dc48-e102-4bec-ba54-d01c5b77bd8a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:15.282464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb92dc48-e102-4bec-ba54-d01c5b77bd8a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:15.393088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1211/2402] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:21:15 2022] POST /identity/v3/users => generated 292 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:21:15.397373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c243e15-7288-4da8-bddc-0b84a4d041f1 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:15.408017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c243e15-7288-4da8-bddc-0b84a4d041f1 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:15.412579 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ae218e-440a-4e21-acb7-968a199b17d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:15.419919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ae218e-440a-4e21-acb7-968a199b17d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.419919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ae218e-440a-4e21-acb7-968a199b17d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.419919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ae218e-440a-4e21-acb7-968a199b17d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.419919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ae218e-440a-4e21-acb7-968a199b17d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:15.419919 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ae218e-440a-4e21-acb7-968a199b17d8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:15.426215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1192/2403] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:21:15.430028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb92dc48-e102-4bec-ba54-d01c5b77bd8a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:15.440448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb92dc48-e102-4bec-ba54-d01c5b77bd8a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:15.444995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94288dc5-8578-44c8-85cd-a477f42dac6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:15.445577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94288dc5-8578-44c8-85cd-a477f42dac6a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.445883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94288dc5-8578-44c8-85cd-a477f42dac6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.446153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94288dc5-8578-44c8-85cd-a477f42dac6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.446781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94288dc5-8578-44c8-85cd-a477f42dac6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:15.448350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94288dc5-8578-44c8-85cd-a477f42dac6a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:15.457983 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1212/2404] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:21:15.461898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ae218e-440a-4e21-acb7-968a199b17d8 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:15.471750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ae218e-440a-4e21-acb7-968a199b17d8 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:15.476208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff0d8c2-5718-4b69-a0c8-420c938a775f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:15.476836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0d8c2-5718-4b69-a0c8-420c938a775f admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.477108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0d8c2-5718-4b69-a0c8-420c938a775f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.477376 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff0d8c2-5718-4b69-a0c8-420c938a775f admin admin] PATH_INFO: `/v3/projects/3ff07ec55bab47a9a1c9fe5904c8140f/users/61e1d88585c549edb61e9980add7ec05/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.492199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff0d8c2-5718-4b69-a0c8-420c938a775f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ff07ec55bab47a9a1c9fe5904c8140f, user_id=61e1d88585c549edb61e9980add7ec05, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:15.494817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff0d8c2-5718-4b69-a0c8-420c938a775f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:15.581365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1193/2405] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:21:15 2022] PUT /identity/v3/projects/3ff07ec55bab47a9a1c9fe5904c8140f/users/61e1d88585c549edb61e9980add7ec05/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:15.591516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6b03e-bd86-456d-b4f0-b6b469fce8cf None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.591800 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6b03e-bd86-456d-b4f0-b6b469fce8cf None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.592073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6b03e-bd86-456d-b4f0-b6b469fce8cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.620811 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0ca6b03e-bd86-456d-b4f0-b6b469fce8cf None None] MFA Rules not processed for user `61e1d88585c549edb61e9980add7ec05`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:15.622069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0ca6b03e-bd86-456d-b4f0-b6b469fce8cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:15.683752 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1213/2406] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:15.688178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff0d8c2-5718-4b69-a0c8-420c938a775f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:15.748234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff0d8c2-5718-4b69-a0c8-420c938a775f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:15.753176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7822bc-29fc-4a76-a1cf-f05262ed0186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:15.753807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7822bc-29fc-4a76-a1cf-f05262ed0186 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.754086 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7822bc-29fc-4a76-a1cf-f05262ed0186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.754359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7822bc-29fc-4a76-a1cf-f05262ed0186 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.755004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7822bc-29fc-4a76-a1cf-f05262ed0186 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:15.756942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7822bc-29fc-4a76-a1cf-f05262ed0186 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:15.921097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1194/2407] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:21:15 2022] POST /identity/v3/projects => generated 381 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:21:15.925106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca6b03e-bd86-456d-b4f0-b6b469fce8cf None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:15.935789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca6b03e-bd86-456d-b4f0-b6b469fce8cf None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:15.991331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a8de89-5893-46a3-930e-e6ef6170ace7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:15.991984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a8de89-5893-46a3-930e-e6ef6170ace7 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:15.992239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a8de89-5893-46a3-930e-e6ef6170ace7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:15.992509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a8de89-5893-46a3-930e-e6ef6170ace7 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:15.993133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a8de89-5893-46a3-930e-e6ef6170ace7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:15.995067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a8de89-5893-46a3-930e-e6ef6170ace7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:16.147428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1214/2408] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:21:15 2022] POST /identity/v3/users => generated 297 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:21:16.151418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7822bc-29fc-4a76-a1cf-f05262ed0186 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:16.161723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7822bc-29fc-4a76-a1cf-f05262ed0186 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:16.166221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30a78de-d2c6-493b-928b-e8851106a994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:16.167327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30a78de-d2c6-493b-928b-e8851106a994 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:16.167604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30a78de-d2c6-493b-928b-e8851106a994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:16.167872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30a78de-d2c6-493b-928b-e8851106a994 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:16.168502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30a78de-d2c6-493b-928b-e8851106a994 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:16.170131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30a78de-d2c6-493b-928b-e8851106a994 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:16.180283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1195/2409] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:21:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:21:16.185030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a8de89-5893-46a3-930e-e6ef6170ace7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:16.195223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a8de89-5893-46a3-930e-e6ef6170ace7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:16.199824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7543c765-8232-4d23-ab78-1abd54de2a2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:16.200463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7543c765-8232-4d23-ab78-1abd54de2a2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:16.200741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7543c765-8232-4d23-ab78-1abd54de2a2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:16.201012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7543c765-8232-4d23-ab78-1abd54de2a2c admin admin] PATH_INFO: `/v3/projects/91e07b83562c472a837645a2c5003eb1/users/17a0ae10face44d4a441477a608c0c7b/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:16.215357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7543c765-8232-4d23-ab78-1abd54de2a2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=91e07b83562c472a837645a2c5003eb1, user_id=17a0ae10face44d4a441477a608c0c7b, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:16.218070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7543c765-8232-4d23-ab78-1abd54de2a2c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:16.347834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1215/2410] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:21:16 2022] PUT /identity/v3/projects/91e07b83562c472a837645a2c5003eb1/users/17a0ae10face44d4a441477a608c0c7b/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:21:16.352047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30a78de-d2c6-493b-928b-e8851106a994 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:16.362388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30a78de-d2c6-493b-928b-e8851106a994 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:16.382343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:16.418157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8212d24c-b028-4544-85a9-ffaec81d63e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:16.418698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212d24c-b028-4544-85a9-ffaec81d63e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:16.418973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212d24c-b028-4544-85a9-ffaec81d63e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:16.419246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8212d24c-b028-4544-85a9-ffaec81d63e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:16.419902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8212d24c-b028-4544-85a9-ffaec81d63e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:16.421547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8212d24c-b028-4544-85a9-ffaec81d63e4 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:16.432050 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1196/2411] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:21:16 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:21:16.436841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7543c765-8232-4d23-ab78-1abd54de2a2c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:16.442174 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:16.447320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7543c765-8232-4d23-ab78-1abd54de2a2c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:16.452209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:16.452859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:16.453141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:16.453414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] PATH_INFO: `/v3/projects/91e07b83562c472a837645a2c5003eb1/users/17a0ae10face44d4a441477a608c0c7b/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:16.455460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91e07b83562c472a837645a2c5003eb1, user_id=17a0ae10face44d4a441477a608c0c7b, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:16.458132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:16.468212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1216/2412] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:21:16 2022] PUT /identity/v3/projects/91e07b83562c472a837645a2c5003eb1/users/17a0ae10face44d4a441477a608c0c7b/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:16.480231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4270dfda-89cc-4eda-9a33-6a0776e9d7a3 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:16.480567 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4270dfda-89cc-4eda-9a33-6a0776e9d7a3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:16.480841 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4270dfda-89cc-4eda-9a33-6a0776e9d7a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:16.508119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4270dfda-89cc-4eda-9a33-6a0776e9d7a3 None None] MFA Rules not processed for user `17a0ae10face44d4a441477a608c0c7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:16.509317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4270dfda-89cc-4eda-9a33-6a0776e9d7a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:16.576032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1197/2413] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:16 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:16.587829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:16.590188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:16.651454 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953e46b4-b4ad-457f-96cb-5182a73e8f43 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:16.655230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb51ca90-007a-45ed-972b-44c9877c60a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:16.655833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb51ca90-007a-45ed-972b-44c9877c60a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:16.656123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb51ca90-007a-45ed-972b-44c9877c60a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:16.656398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb51ca90-007a-45ed-972b-44c9877c60a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:16.665949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb51ca90-007a-45ed-972b-44c9877c60a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:16.667477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb51ca90-007a-45ed-972b-44c9877c60a8 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:16.676886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1217/2414] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:21:16 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:21:16.858742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4270dfda-89cc-4eda-9a33-6a0776e9d7a3 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:16.868295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4270dfda-89cc-4eda-9a33-6a0776e9d7a3 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:16.872326 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcabfdd-cfb8-429e-b5ea-1d33e4542049 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:16.872905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcabfdd-cfb8-429e-b5ea-1d33e4542049 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:16.873190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcabfdd-cfb8-429e-b5ea-1d33e4542049 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:16.873462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcabfdd-cfb8-429e-b5ea-1d33e4542049 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:16.928959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcabfdd-cfb8-429e-b5ea-1d33e4542049 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:16.932140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcabfdd-cfb8-429e-b5ea-1d33e4542049 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:16.941443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1198/2415] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:21:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:21:17.446818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb51ca90-007a-45ed-972b-44c9877c60a8 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:17.507575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb51ca90-007a-45ed-972b-44c9877c60a8 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:17.512474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:17.513100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:17.513379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:17.513680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] PATH_INFO: `/v3/users/61e1d88585c549edb61e9980add7ec05` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:17.514875 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=61e1d88585c549edb61e9980add7ec05)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:17.516825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:17.709213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61e1d88585c549edb61e9980add7ec05'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:17.709884 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61e1d88585c549edb61e9980add7ec05'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:17.721806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61e1d88585c549edb61e9980add7ec05'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:17.804450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61e1d88585c549edb61e9980add7ec05'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:17.812149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61e1d88585c549edb61e9980add7ec05'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:17.817967 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1218/2416] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:21:17 2022] DELETE /identity/v3/users/61e1d88585c549edb61e9980add7ec05 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:17.822605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d6eff-208b-4c9d-8701-7fc4b2b63bdb None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:17.822892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d6eff-208b-4c9d-8701-7fc4b2b63bdb None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:17.823165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d6eff-208b-4c9d-8701-7fc4b2b63bdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:17.841312 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-202d6eff-208b-4c9d-8701-7fc4b2b63bdb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:17.842529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-202d6eff-208b-4c9d-8701-7fc4b2b63bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:17.882670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:17.900574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1199/2417] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:17.910084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:17.911455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:17.925233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:17.967391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f658aa-b1eb-4705-b507-b560ecd67b78 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:17.970990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da72b96-d7a7-4031-9526-9c620d60f5b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:17.971847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da72b96-d7a7-4031-9526-9c620d60f5b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:17.972119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da72b96-d7a7-4031-9526-9c620d60f5b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:17.972387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da72b96-d7a7-4031-9526-9c620d60f5b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:17.981638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da72b96-d7a7-4031-9526-9c620d60f5b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:17.983045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da72b96-d7a7-4031-9526-9c620d60f5b6 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:17.992218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1219/2418] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:21:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:21:18.428553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202d6eff-208b-4c9d-8701-7fc4b2b63bdb None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:18.430039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-202d6eff-208b-4c9d-8701-7fc4b2b63bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:18.442067 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202d6eff-208b-4c9d-8701-7fc4b2b63bdb None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:18.446722 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:18.447385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:18.447664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:18.448429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] PATH_INFO: `/v3/projects/3ff07ec55bab47a9a1c9fe5904c8140f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:18.449548 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ff07ec55bab47a9a1c9fe5904c8140f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:18.451655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:18.599476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ff07ec55bab47a9a1c9fe5904c8140f', 'request_id': 'req-776b2caa-20d7-4820-b20c-6294cb984fe1'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:18.690251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ff07ec55bab47a9a1c9fe5904c8140f', 'request_id': 'req-776b2caa-20d7-4820-b20c-6294cb984fe1'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:18.696072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ff07ec55bab47a9a1c9fe5904c8140f', 'request_id': 'req-776b2caa-20d7-4820-b20c-6294cb984fe1'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:18.707594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] The token cache is being invalidate because project 3ff07ec55bab47a9a1c9fe5904c8140f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:21:18.707866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:18.708500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:18.710015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1200/2419] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:21:18 2022] DELETE /identity/v3/projects/3ff07ec55bab47a9a1c9fe5904c8140f => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:18.714055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da72b96-d7a7-4031-9526-9c620d60f5b6 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:18.724469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da72b96-d7a7-4031-9526-9c620d60f5b6 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:18.780637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:18.781259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:18.781538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:18.781841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] PATH_INFO: `/v3/users/17a0ae10face44d4a441477a608c0c7b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:18.782939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] RBAC: Authorizing `identity:delete_user(user_id=17a0ae10face44d4a441477a608c0c7b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:18.784846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:18.983696 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17a0ae10face44d4a441477a608c0c7b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:18.984267 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17a0ae10face44d4a441477a608c0c7b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:18.996816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17a0ae10face44d4a441477a608c0c7b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:19.095542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17a0ae10face44d4a441477a608c0c7b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:19.102872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f03d8202-867e-43de-83dd-2de88baf48af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17a0ae10face44d4a441477a608c0c7b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:19.108764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1220/2420] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:21:18 2022] DELETE /identity/v3/users/17a0ae10face44d4a441477a608c0c7b => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:21:19.617692 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:19.619133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:19.632826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776b2caa-20d7-4820-b20c-6294cb984fe1 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:19.689216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:19.689867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:19.690142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:19.690408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] PATH_INFO: `/v3/projects/91e07b83562c472a837645a2c5003eb1` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:19.691503 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91e07b83562c472a837645a2c5003eb1)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:19.693377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:19.839409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91e07b83562c472a837645a2c5003eb1', 'request_id': 'req-fb51f29f-1202-4a3d-9a40-78d4e47083b0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:19.924220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91e07b83562c472a837645a2c5003eb1', 'request_id': 'req-fb51f29f-1202-4a3d-9a40-78d4e47083b0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:19.930037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91e07b83562c472a837645a2c5003eb1', 'request_id': 'req-fb51f29f-1202-4a3d-9a40-78d4e47083b0'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:19.937031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:19.941394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] The token cache is being invalidate because project 91e07b83562c472a837645a2c5003eb1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:21:19.941716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:19.942336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-fb51f29f-1202-4a3d-9a40-78d4e47083b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:19.943785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1201/2421] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:21:19 2022] DELETE /identity/v3/projects/91e07b83562c472a837645a2c5003eb1 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:19.949156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763542a6-4101-400c-ba8c-dc6b8a83d604 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:19.949641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763542a6-4101-400c-ba8c-dc6b8a83d604 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:19.949759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763542a6-4101-400c-ba8c-dc6b8a83d604 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:19.975244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-763542a6-4101-400c-ba8c-dc6b8a83d604 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:19.976426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-763542a6-4101-400c-ba8c-dc6b8a83d604 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:20.004975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:20.035067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1221/2422] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:20.043063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6446c2f2-cce2-487c-b49d-8d412a0995ef None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.043361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6446c2f2-cce2-487c-b49d-8d412a0995ef None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.043639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6446c2f2-cce2-487c-b49d-8d412a0995ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.062764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6446c2f2-cce2-487c-b49d-8d412a0995ef None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:20.065351 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6446c2f2-cce2-487c-b49d-8d412a0995ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:20.071991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1202/2423] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:20.076581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763542a6-4101-400c-ba8c-dc6b8a83d604 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:20.086695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763542a6-4101-400c-ba8c-dc6b8a83d604 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:20.091932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:20.094029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.094029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.094029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.094029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:21:20.094029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:20.095686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:20.107243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1222/2424] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:21:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:21:20.111338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6446c2f2-cce2-487c-b49d-8d412a0995ef None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:20.120669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6446c2f2-cce2-487c-b49d-8d412a0995ef None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:20.125653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512a868-d741-4ec5-bc30-f05f39141563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:20.128449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a868-d741-4ec5-bc30-f05f39141563 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.128449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a868-d741-4ec5-bc30-f05f39141563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.128449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a868-d741-4ec5-bc30-f05f39141563 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.128449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b512a868-d741-4ec5-bc30-f05f39141563 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:20.129402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b512a868-d741-4ec5-bc30-f05f39141563 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:20.274745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1203/2425] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:21:20 2022] POST /identity/v3/projects => generated 373 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:21:20.279177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:20.289254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90b3b8-a883-4671-9662-e745d1ba7ad8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:20.344366 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f7c6a2-4d7a-464b-a23e-1cba6178b36d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:20.344969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f7c6a2-4d7a-464b-a23e-1cba6178b36d admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.345249 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f7c6a2-4d7a-464b-a23e-1cba6178b36d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.345546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f7c6a2-4d7a-464b-a23e-1cba6178b36d admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.346198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f7c6a2-4d7a-464b-a23e-1cba6178b36d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:20.348389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f7c6a2-4d7a-464b-a23e-1cba6178b36d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:20.456078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1223/2426] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:21:20 2022] POST /identity/v3/users => generated 287 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:21:20.460130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512a868-d741-4ec5-bc30-f05f39141563 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:20.471069 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512a868-d741-4ec5-bc30-f05f39141563 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:20.476278 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9249e2-ad23-4387-9bc1-44bdf415147a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:20.476910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9249e2-ad23-4387-9bc1-44bdf415147a admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.477187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9249e2-ad23-4387-9bc1-44bdf415147a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.477564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9249e2-ad23-4387-9bc1-44bdf415147a admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.478240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9249e2-ad23-4387-9bc1-44bdf415147a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:20.479834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9249e2-ad23-4387-9bc1-44bdf415147a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:20.489980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1204/2427] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:21:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:21:20.493850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f7c6a2-4d7a-464b-a23e-1cba6178b36d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:20.504982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f7c6a2-4d7a-464b-a23e-1cba6178b36d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:20.509899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1dcb02-082f-47b5-ad28-b5b79cb7a258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:20.511124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1dcb02-082f-47b5-ad28-b5b79cb7a258 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.511397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1dcb02-082f-47b5-ad28-b5b79cb7a258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.511694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1dcb02-082f-47b5-ad28-b5b79cb7a258 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.512323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1dcb02-082f-47b5-ad28-b5b79cb7a258 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:20.513987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1dcb02-082f-47b5-ad28-b5b79cb7a258 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:20.524279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1224/2428] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:21:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:21:20.528238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9249e2-ad23-4387-9bc1-44bdf415147a admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:20.538011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9249e2-ad23-4387-9bc1-44bdf415147a admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:20.542489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:20.543132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.543405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.543674 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] PATH_INFO: `/v3/projects/b74abfdbf2414fc59b2938f17c402bf8/users/6491f5f0f6eb4d9485eec16b5a7cf9ca/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.558635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b74abfdbf2414fc59b2938f17c402bf8, user_id=6491f5f0f6eb4d9485eec16b5a7cf9ca, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:20.561834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:20.621808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1205/2429] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:21:20 2022] PUT /identity/v3/projects/b74abfdbf2414fc59b2938f17c402bf8/users/6491f5f0f6eb4d9485eec16b5a7cf9ca/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:20.633324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c800b569-fbea-4c4f-b464-585d804db628 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.633636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c800b569-fbea-4c4f-b464-585d804db628 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.633915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c800b569-fbea-4c4f-b464-585d804db628 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.661389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c800b569-fbea-4c4f-b464-585d804db628 None None] MFA Rules not processed for user `6491f5f0f6eb4d9485eec16b5a7cf9ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:20.662605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c800b569-fbea-4c4f-b464-585d804db628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:20.723789 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1225/2430] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:20.734737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:20.736413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:20.792157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c45a1c-6628-4425-b453-db2543632b5a admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:20.795801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c61d5b-9218-4743-8595-20fe9c259263 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:20.796427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c61d5b-9218-4743-8595-20fe9c259263 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:20.796740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c61d5b-9218-4743-8595-20fe9c259263 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:20.797010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c61d5b-9218-4743-8595-20fe9c259263 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:20.805174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c61d5b-9218-4743-8595-20fe9c259263 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:20.806640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c61d5b-9218-4743-8595-20fe9c259263 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:20.815924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1206/2431] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:21:20 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:21:30.720674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=84,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:30.811913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=54,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:33.092195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c800b569-fbea-4c4f-b464-585d804db628 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:33.153775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c800b569-fbea-4c4f-b464-585d804db628 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:33.158166 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:33.158781 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:33.159057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:33.159325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] PATH_INFO: `/v3/users/6491f5f0f6eb4d9485eec16b5a7cf9ca` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:33.160442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6491f5f0f6eb4d9485eec16b5a7cf9ca)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:33.162377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:33.325849 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6491f5f0f6eb4d9485eec16b5a7cf9ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:33.326432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6491f5f0f6eb4d9485eec16b5a7cf9ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:33.338621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6491f5f0f6eb4d9485eec16b5a7cf9ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:33.387149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6491f5f0f6eb4d9485eec16b5a7cf9ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:33.394203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6491f5f0f6eb4d9485eec16b5a7cf9ca'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:33.400004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1226/2432] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:21:33 2022] DELETE /identity/v3/users/6491f5f0f6eb4d9485eec16b5a7cf9ca => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:33.405120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d53471-f468-4a0b-9e85-25b5daa66e6f None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:33.405621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d53471-f468-4a0b-9e85-25b5daa66e6f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:33.405728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d53471-f468-4a0b-9e85-25b5daa66e6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:33.423694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-76d53471-f468-4a0b-9e85-25b5daa66e6f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:33.424871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-76d53471-f468-4a0b-9e85-25b5daa66e6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:33.483858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1207/2433] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:33.497794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:33.497794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:33.569364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3e6a50-a87e-4a64-a037-a65ee6048760 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:33.573716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b126f235-61c8-4989-889e-cfcb292e4be4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:33.574386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b126f235-61c8-4989-889e-cfcb292e4be4 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:33.574667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b126f235-61c8-4989-889e-cfcb292e4be4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:33.574940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b126f235-61c8-4989-889e-cfcb292e4be4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:33.584100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b126f235-61c8-4989-889e-cfcb292e4be4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:33.585502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b126f235-61c8-4989-889e-cfcb292e4be4 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:33.595568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1227/2434] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:21:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:21:34.012563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d53471-f468-4a0b-9e85-25b5daa66e6f None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:34.014000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-76d53471-f468-4a0b-9e85-25b5daa66e6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:34.025694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d53471-f468-4a0b-9e85-25b5daa66e6f None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:34.030451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:34.031977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:34.032266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:34.032535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] PATH_INFO: `/v3/projects/b74abfdbf2414fc59b2938f17c402bf8` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:34.034101 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b74abfdbf2414fc59b2938f17c402bf8)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:34.036020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:34.158766 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b74abfdbf2414fc59b2938f17c402bf8', 'request_id': 'req-71487d3a-16ac-44c3-80a5-85b56ec69c3e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:34.547224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b74abfdbf2414fc59b2938f17c402bf8', 'request_id': 'req-71487d3a-16ac-44c3-80a5-85b56ec69c3e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:34.553509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b74abfdbf2414fc59b2938f17c402bf8', 'request_id': 'req-71487d3a-16ac-44c3-80a5-85b56ec69c3e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:34.564884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] The token cache is being invalidate because project b74abfdbf2414fc59b2938f17c402bf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:21:34.565172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:34.565835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-71487d3a-16ac-44c3-80a5-85b56ec69c3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:21:34.567366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1208/2435] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:21:34 2022] DELETE /identity/v3/projects/b74abfdbf2414fc59b2938f17c402bf8 => generated 0 bytes in 555 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:34.572076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3c851-901a-4c4e-9bba-7802209e8f01 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:34.572527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3c851-901a-4c4e-9bba-7802209e8f01 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:34.572635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d3c851-901a-4c4e-9bba-7802209e8f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:34.582403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:34.620762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-68d3c851-901a-4c4e-9bba-7802209e8f01 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:34.621971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-68d3c851-901a-4c4e-9bba-7802209e8f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:34.679055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1228/2436] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:34.687202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc0748-15a8-42b4-9c83-9860487e3f22 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:34.687557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc0748-15a8-42b4-9c83-9860487e3f22 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:34.687864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc0748-15a8-42b4-9c83-9860487e3f22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:34.707536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3fcc0748-15a8-42b4-9c83-9860487e3f22 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:34.708729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3fcc0748-15a8-42b4-9c83-9860487e3f22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:34.716370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1209/2437] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:34.720709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3c851-901a-4c4e-9bba-7802209e8f01 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:34.730540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d3c851-901a-4c4e-9bba-7802209e8f01 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:34.735049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:34.735645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:34.735926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:34.736198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:34.736838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:21:34.737140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:34.738762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:34.751021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1229/2438] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:21:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:21:34.755320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc0748-15a8-42b4-9c83-9860487e3f22 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:34.765365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc0748-15a8-42b4-9c83-9860487e3f22 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:34.769961 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363af942-50a5-43f3-9fd0-b781ca268d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:34.770571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363af942-50a5-43f3-9fd0-b781ca268d92 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:34.770847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363af942-50a5-43f3-9fd0-b781ca268d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:34.771115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363af942-50a5-43f3-9fd0-b781ca268d92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:34.771733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363af942-50a5-43f3-9fd0-b781ca268d92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:34.773623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363af942-50a5-43f3-9fd0-b781ca268d92 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:34.833836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1210/2439] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:21:34 2022] POST /identity/v3/projects => generated 389 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:21:34.837978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:34.848444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4da6250-01dc-4894-b0d9-a0a4998ba2ce admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:34.902863 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab194a2-7d1f-44bf-9206-f7eef2b67254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:34.903456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab194a2-7d1f-44bf-9206-f7eef2b67254 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:34.903733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab194a2-7d1f-44bf-9206-f7eef2b67254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:34.904003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab194a2-7d1f-44bf-9206-f7eef2b67254 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:34.904613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab194a2-7d1f-44bf-9206-f7eef2b67254 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:34.906544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab194a2-7d1f-44bf-9206-f7eef2b67254 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:35.013408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1230/2440] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:21:34 2022] POST /identity/v3/users => generated 295 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:21:35.014743 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363af942-50a5-43f3-9fd0-b781ca268d92 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:35.024930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363af942-50a5-43f3-9fd0-b781ca268d92 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:35.029708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c932c1-a583-4cd1-bd09-4f36e5ccb149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:35.030289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c932c1-a583-4cd1-bd09-4f36e5ccb149 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:35.030566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c932c1-a583-4cd1-bd09-4f36e5ccb149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:35.030835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c932c1-a583-4cd1-bd09-4f36e5ccb149 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:35.031465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c932c1-a583-4cd1-bd09-4f36e5ccb149 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:35.033056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c932c1-a583-4cd1-bd09-4f36e5ccb149 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:35.043354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1211/2441] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:21:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:21:35.047171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab194a2-7d1f-44bf-9206-f7eef2b67254 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:35.057479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab194a2-7d1f-44bf-9206-f7eef2b67254 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:35.062100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a662a23-8c7c-4e40-875f-e3cedee3f07b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:35.062688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a662a23-8c7c-4e40-875f-e3cedee3f07b admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:35.062968 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a662a23-8c7c-4e40-875f-e3cedee3f07b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:35.063240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a662a23-8c7c-4e40-875f-e3cedee3f07b admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:35.063877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a662a23-8c7c-4e40-875f-e3cedee3f07b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:35.065447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a662a23-8c7c-4e40-875f-e3cedee3f07b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:35.076556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1231/2442] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:21:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:21:35.080559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c932c1-a583-4cd1-bd09-4f36e5ccb149 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:35.090747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c932c1-a583-4cd1-bd09-4f36e5ccb149 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:35.095310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:35.095954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:35.096243 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:35.096508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] PATH_INFO: `/v3/projects/981003711acd4115912320529b2d49cf/users/ca0d973bcd18479cb344b1624980c08b/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:35.112733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] RBAC: Authorizing `identity:create_grant(project_id=981003711acd4115912320529b2d49cf, user_id=ca0d973bcd18479cb344b1624980c08b, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:35.115897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:35.221242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1212/2443] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:21:35 2022] PUT /identity/v3/projects/981003711acd4115912320529b2d49cf/users/ca0d973bcd18479cb344b1624980c08b/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:21:35.232577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7eeb9-4a2f-4d14-a65f-a8a13059ec5f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:35.235816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7eeb9-4a2f-4d14-a65f-a8a13059ec5f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:35.235816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f7eeb9-4a2f-4d14-a65f-a8a13059ec5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:35.266077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-20f7eeb9-4a2f-4d14-a65f-a8a13059ec5f None None] MFA Rules not processed for user `ca0d973bcd18479cb344b1624980c08b`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:35.273582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-20f7eeb9-4a2f-4d14-a65f-a8a13059ec5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:35.306400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:35.330910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1232/2444] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:35 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:21:35.341999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:21:35.343851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:35.366215 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:35.401338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2ef98e-5e7b-4744-986f-2fb649f84775 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:21:35.404948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:21:35.405528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:35.405833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:35.406104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:35.414415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:21:35.415824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:21:35.424947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1213/2445] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:21:35 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:21:45.381444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:45.478192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=17 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:21:54.691493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e065031d-2bfa-41e4-b4e4-70b53a93e331 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:21:54.692007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e065031d-2bfa-41e4-b4e4-70b53a93e331 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:21:54.692007 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e065031d-2bfa-41e4-b4e4-70b53a93e331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:21:54.711616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e065031d-2bfa-41e4-b4e4-70b53a93e331 None None] MFA Rules not processed for user `ec39d995899c4d43bb4f8ffd9e6d2ef3`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:21:54.712816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e065031d-2bfa-41e4-b4e4-70b53a93e331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:21:54.767567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1233/2446] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:21:54 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:22:01.825789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:01.831609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:01.932591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ce7969-3dcc-4fd8-8787-1002bbda58e7 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:01.939627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fa2072-374b-489d-9e8b-e52e5c749d94 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:01.940119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fa2072-374b-489d-9e8b-e52e5c749d94 service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:01.940119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fa2072-374b-489d-9e8b-e52e5c749d94 service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:01.940404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fa2072-374b-489d-9e8b-e52e5c749d94 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:01.942460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d9fa2072-374b-489d-9e8b-e52e5c749d94 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:01.969136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fa2072-374b-489d-9e8b-e52e5c749d94 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:01.969343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fa2072-374b-489d-9e8b-e52e5c749d94 service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:01.970240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e065031d-2bfa-41e4-b4e4-70b53a93e331 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:01.981443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1214/2447] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:22:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:22:02.038969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e065031d-2bfa-41e4-b4e4-70b53a93e331 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:02.043596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:02.044288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:02.044653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:02.044933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] PATH_INFO: `/v3/users/ca0d973bcd18479cb344b1624980c08b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:02.046104 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca0d973bcd18479cb344b1624980c08b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:02.048040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:02.332754 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca0d973bcd18479cb344b1624980c08b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:02.333417 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca0d973bcd18479cb344b1624980c08b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:02.345193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca0d973bcd18479cb344b1624980c08b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:02.415669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca0d973bcd18479cb344b1624980c08b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:02.423434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca0d973bcd18479cb344b1624980c08b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:02.429304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1234/2448] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:22:01 2022] DELETE /identity/v3/users/ca0d973bcd18479cb344b1624980c08b => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:22:02.433906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248b32c-cc6b-4b1c-81fc-73343d5c21fb None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:02.434269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248b32c-cc6b-4b1c-81fc-73343d5c21fb None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:02.434613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5248b32c-cc6b-4b1c-81fc-73343d5c21fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:02.454476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5248b32c-cc6b-4b1c-81fc-73343d5c21fb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:22:02.455676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5248b32c-cc6b-4b1c-81fc-73343d5c21fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:02.514636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1215/2449] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:22:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:22:02.524237 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:02.525676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:02.553671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:22:02.605425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9c6ea-416d-45ae-b07c-42965d17f7c1 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:02.609508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a644d4-47a0-4e57-be97-1e7f713432a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:02.610136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a644d4-47a0-4e57-be97-1e7f713432a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:02.610413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a644d4-47a0-4e57-be97-1e7f713432a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:02.610677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a644d4-47a0-4e57-be97-1e7f713432a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:02.619571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a644d4-47a0-4e57-be97-1e7f713432a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:02.620985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a644d4-47a0-4e57-be97-1e7f713432a5 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:02.631443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1235/2450] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:22:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:22:03.070981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248b32c-cc6b-4b1c-81fc-73343d5c21fb None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:03.072415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5248b32c-cc6b-4b1c-81fc-73343d5c21fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:03.084300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5248b32c-cc6b-4b1c-81fc-73343d5c21fb None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:03.090806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:03.091443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:03.091784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:03.092055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] PATH_INFO: `/v3/projects/981003711acd4115912320529b2d49cf` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:03.093777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=981003711acd4115912320529b2d49cf)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:03.095916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:03.242075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '981003711acd4115912320529b2d49cf', 'request_id': 'req-b94965dd-780f-4374-8fb8-35299d6f66c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:03.307171 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '981003711acd4115912320529b2d49cf', 'request_id': 'req-b94965dd-780f-4374-8fb8-35299d6f66c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:03.317140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '981003711acd4115912320529b2d49cf', 'request_id': 'req-b94965dd-780f-4374-8fb8-35299d6f66c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:03.318796 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:22:03.331023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] The token cache is being invalidate because project 981003711acd4115912320529b2d49cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:22:03.331463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:03.332518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:22:03.334393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1216/2451] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:22:03 2022] DELETE /identity/v3/projects/981003711acd4115912320529b2d49cf => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:22:09.587738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa1e81b-79d8-451d-99d3-c371e6224786 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:09.588212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa1e81b-79d8-451d-99d3-c371e6224786 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:09.588303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa1e81b-79d8-451d-99d3-c371e6224786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:09.618960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ffa1e81b-79d8-451d-99d3-c371e6224786 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:22:09.620210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ffa1e81b-79d8-451d-99d3-c371e6224786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:09.679812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1236/2452] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:22:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:22:09.818600 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:09.819993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:09.887976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94965dd-780f-4374-8fb8-35299d6f66c5 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:09.893428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56b7610-4006-45b1-a0cd-aa8985786309 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:09.894083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56b7610-4006-45b1-a0cd-aa8985786309 service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:09.894481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56b7610-4006-45b1-a0cd-aa8985786309 service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:09.895154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56b7610-4006-45b1-a0cd-aa8985786309 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:09.902858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56b7610-4006-45b1-a0cd-aa8985786309 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:09.904470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56b7610-4006-45b1-a0cd-aa8985786309 service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:09.912932 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1217/2453] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:22:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:22:19.672922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=56 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:22:19.909787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=35 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:22:27.531926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047cf03a-80d4-4342-9a3b-260517d20f5f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:27.532396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047cf03a-80d4-4342-9a3b-260517d20f5f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:27.532503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047cf03a-80d4-4342-9a3b-260517d20f5f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:27.533879 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1237/2454] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:22:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:22:27.539575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fc13a9-63ac-4a23-b28b-3d4a72c5a90a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:27.540042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fc13a9-63ac-4a23-b28b-3d4a72c5a90a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:27.540144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fc13a9-63ac-4a23-b28b-3d4a72c5a90a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:27.572680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-09fc13a9-63ac-4a23-b28b-3d4a72c5a90a None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:22:27.573957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-09fc13a9-63ac-4a23-b28b-3d4a72c5a90a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:27.650455 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1218/2455] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:22:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:22:27.662551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047cf03a-80d4-4342-9a3b-260517d20f5f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:27.664182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-047cf03a-80d4-4342-9a3b-260517d20f5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:27.721013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047cf03a-80d4-4342-9a3b-260517d20f5f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:27.724653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202d38b3-ca30-44c9-9dc6-0ba242ae33b2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6cc8f34f9674a94ba23a7157e44a6f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:27.725243 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d38b3-ca30-44c9-9dc6-0ba242ae33b2 service glance] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:27.725517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d38b3-ca30-44c9-9dc6-0ba242ae33b2 service glance] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:27.725818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202d38b3-ca30-44c9-9dc6-0ba242ae33b2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:27.736058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202d38b3-ca30-44c9-9dc6-0ba242ae33b2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:27.737457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202d38b3-ca30-44c9-9dc6-0ba242ae33b2 service glance] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:27.747712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1238/2456] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:22:27 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:22:27.982511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4a4ed3-9099-44c3-a6d0-60370e2e9b70 None None] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:27.982733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4a4ed3-9099-44c3-a6d0-60370e2e9b70 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:27.983009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4a4ed3-9099-44c3-a6d0-60370e2e9b70 None None] PATH_INFO: `/` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:27.984450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1219/2457] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:22:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:22:27.989263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c873ab1-05a2-4d1c-828b-cd16765fc23d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:27.989569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c873ab1-05a2-4d1c-828b-cd16765fc23d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:27.994153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c873ab1-05a2-4d1c-828b-cd16765fc23d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:28.009167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5c873ab1-05a2-4d1c-828b-cd16765fc23d None None] MFA Rules not processed for user `9220ba3637a542a382ad1d42ff1ff459`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:22:28.010409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5c873ab1-05a2-4d1c-828b-cd16765fc23d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:28.061748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1239/2458] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:22:27 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:22:28.073604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4a4ed3-9099-44c3-a6d0-60370e2e9b70 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:28.075314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3d4a4ed3-9099-44c3-a6d0-60370e2e9b70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:28.089252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4a4ed3-9099-44c3-a6d0-60370e2e9b70 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:28.092840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b66480-e4cb-4818-8e5e-ec2d0754e72f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:28.094085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b66480-e4cb-4818-8e5e-ec2d0754e72f service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:28.094368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b66480-e4cb-4818-8e5e-ec2d0754e72f service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:28.094640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b66480-e4cb-4818-8e5e-ec2d0754e72f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:28.101941 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b66480-e4cb-4818-8e5e-ec2d0754e72f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:28.103353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b66480-e4cb-4818-8e5e-ec2d0754e72f service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:28.111108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1220/2459] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:22:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:22:28.197718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b353374-6960-401e-adfe-cbf3498b652f None None] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:28.200389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b353374-6960-401e-adfe-cbf3498b652f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:28.200389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b353374-6960-401e-adfe-cbf3498b652f None None] PATH_INFO: `/` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:28.200389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1240/2460] 10.222.0.16 () {54 vars in 921 bytes} [Thu Aug 25 05:22:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 25 05:22:28.205213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558fb41b-030e-426d-a844-ca66e6d04182 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:28.205555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558fb41b-030e-426d-a844-ca66e6d04182 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:28.205861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558fb41b-030e-426d-a844-ca66e6d04182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:28.226452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-558fb41b-030e-426d-a844-ca66e6d04182 None None] MFA Rules not processed for user `eb5ac6ef2f5048d3aed0d7a91abc4cac`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:22:28.227651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-558fb41b-030e-426d-a844-ca66e6d04182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:28.237063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1221/2461] 10.222.0.16 () {58 vars in 1018 bytes} [Thu Aug 25 05:22:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:22:28.247347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b353374-6960-401e-adfe-cbf3498b652f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:28.258223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b353374-6960-401e-adfe-cbf3498b652f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:28.261778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f158dd9b-75d8-4d74-9494-592d8f41fa8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:28.262376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158dd9b-75d8-4d74-9494-592d8f41fa8c service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:28.262658 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158dd9b-75d8-4d74-9494-592d8f41fa8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:28.262933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f158dd9b-75d8-4d74-9494-592d8f41fa8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:28.273311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f158dd9b-75d8-4d74-9494-592d8f41fa8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:28.274766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f158dd9b-75d8-4d74-9494-592d8f41fa8c service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:28.286228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1241/2462] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:22:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:22:30.792709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb2124-3cff-4a1e-9020-894de00f61bf None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:30.792709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb2124-3cff-4a1e-9020-894de00f61bf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:30.792709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb2124-3cff-4a1e-9020-894de00f61bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:30.792709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-82cb2124-3cff-4a1e-9020-894de00f61bf None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:22:30.792709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-82cb2124-3cff-4a1e-9020-894de00f61bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:30.792709 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1222/2463] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:22:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:22:30.794089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f158dd9b-75d8-4d74-9494-592d8f41fa8c service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:30.794089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f158dd9b-75d8-4d74-9494-592d8f41fa8c service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:30.794089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4708d9-4ee0-498b-9361-366815f5ea5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:30.794089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4708d9-4ee0-498b-9361-366815f5ea5d service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:30.794850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4708d9-4ee0-498b-9361-366815f5ea5d service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:30.794850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4708d9-4ee0-498b-9361-366815f5ea5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:30.794850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4708d9-4ee0-498b-9361-366815f5ea5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:30.794850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4708d9-4ee0-498b-9361-366815f5ea5d service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:30.794850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1242/2464] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:22:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:22:40.726323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=52 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:22:40.787450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=61 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:22:54.096377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022ef05-fa6d-45ec-8530-51ed51d12224 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:54.096805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022ef05-fa6d-45ec-8530-51ed51d12224 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:54.096925 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3022ef05-fa6d-45ec-8530-51ed51d12224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:54.115499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3022ef05-fa6d-45ec-8530-51ed51d12224 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:22:54.116668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3022ef05-fa6d-45ec-8530-51ed51d12224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:22:54.123524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1223/2465] 10.222.0.16 () {58 vars in 967 bytes} [Thu Aug 25 05:22:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:22:54.135911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4708d9-4ee0-498b-9361-366815f5ea5d service swift] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:22:54.147133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4708d9-4ee0-498b-9361-366815f5ea5d service swift] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:22:54.152254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:22:54.152881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:22:54.153182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:22:54.153453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:22:54.161507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:22:54.163106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:22:54.170940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1243/2466] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:22:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:23:04.110957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:04.167727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:22.679552 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3022ef05-fa6d-45ec-8530-51ed51d12224 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:22.689291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3022ef05-fa6d-45ec-8530-51ed51d12224 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:22.692797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:22.693387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:22.693697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:22.693970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:22.695672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:22.756959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:22.758400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:22.767781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1224/2467] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:26.198890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:26.200455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:26.271528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6605b30a-beca-457a-8e43-8204c504a608 service swift] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:26.276560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:26.277196 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:26.277476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:26.277786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:26.279493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:26.291596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:26.293217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:26.303883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1244/2468] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:26.424933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:26.426428 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:26.438737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131596a4-6db4-4ba0-9222-0e36c854ce41 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:26.443421 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:26.444043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:26.444320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:26.444588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] PATH_INFO: `/v3/users/d61cf1670a7a43519c7258cd467227b6` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:26.458440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d61cf1670a7a43519c7258cd467227b6)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:26.460377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:26.719879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd61cf1670a7a43519c7258cd467227b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:26.731850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd61cf1670a7a43519c7258cd467227b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:26.732508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd61cf1670a7a43519c7258cd467227b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:26.826020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd61cf1670a7a43519c7258cd467227b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:26.833767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b0744039-6c45-4634-a41f-66ab4226dc16 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd61cf1670a7a43519c7258cd467227b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:26.839770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1225/2469] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:23:26 2022] DELETE /identity/v3/users/d61cf1670a7a43519c7258cd467227b6 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:27.082369 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:27.094847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108907c3-4900-423f-b206-c7ef11be9bde service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:27.159865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:27.160491 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.160768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.161099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] PATH_INFO: `/v3/projects/2eb9cdd7b67546ca8eb089247f0a0ce1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.171208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2eb9cdd7b67546ca8eb089247f0a0ce1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:27.173659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:27.288787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2eb9cdd7b67546ca8eb089247f0a0ce1', 'request_id': 'req-2589b777-3217-48c4-9d81-3484b383091e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:27.294280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2eb9cdd7b67546ca8eb089247f0a0ce1', 'request_id': 'req-2589b777-3217-48c4-9d81-3484b383091e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:27.391674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2eb9cdd7b67546ca8eb089247f0a0ce1', 'request_id': 'req-2589b777-3217-48c4-9d81-3484b383091e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:27.404766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] The token cache is being invalidate because project 2eb9cdd7b67546ca8eb089247f0a0ce1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:23:27.405057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:27.405722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2589b777-3217-48c4-9d81-3484b383091e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:27.407179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1245/2470] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:23:27 2022] DELETE /identity/v3/projects/2eb9cdd7b67546ca8eb089247f0a0ce1 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:27.412218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f10986-2921-4f0c-a1b8-e5aad6410ef2 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.412560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f10986-2921-4f0c-a1b8-e5aad6410ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.412840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f10986-2921-4f0c-a1b8-e5aad6410ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.433058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-a2f10986-2921-4f0c-a1b8-e5aad6410ef2 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:27.434276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a2f10986-2921-4f0c-a1b8-e5aad6410ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:27.481792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:27.493656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1226/2471] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:27.502440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87727b50-9ef7-4475-a5a4-9f29e1a990f1 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.502720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87727b50-9ef7-4475-a5a4-9f29e1a990f1 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.502991 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87727b50-9ef7-4475-a5a4-9f29e1a990f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.523490 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-87727b50-9ef7-4475-a5a4-9f29e1a990f1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:27.527389 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-87727b50-9ef7-4475-a5a4-9f29e1a990f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:27.533129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1246/2472] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:27.537309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f10986-2921-4f0c-a1b8-e5aad6410ef2 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:27.547550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f10986-2921-4f0c-a1b8-e5aad6410ef2 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:27.552613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:27.553204 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.553479 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.553781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.554412 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:23:27.554710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:27.556273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:27.568057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1227/2473] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:23:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:23:27.576169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87727b50-9ef7-4475-a5a4-9f29e1a990f1 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:27.581839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:27.585668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87727b50-9ef7-4475-a5a4-9f29e1a990f1 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:27.590435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089c0d3-1674-41c3-b8d3-e2ab6a1e5e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:27.591031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089c0d3-1674-41c3-b8d3-e2ab6a1e5e79 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.591307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089c0d3-1674-41c3-b8d3-e2ab6a1e5e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.591576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089c0d3-1674-41c3-b8d3-e2ab6a1e5e79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.592207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8089c0d3-1674-41c3-b8d3-e2ab6a1e5e79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:27.594669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8089c0d3-1674-41c3-b8d3-e2ab6a1e5e79 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:27.695382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1247/2474] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:23:27 2022] POST /identity/v3/projects => generated 385 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:23:27.699690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:27.711664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0166d98c-7a3b-46da-b473-0e360496c89b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:27.765874 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb92fa3-edcd-487e-8884-528a6d94980e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:27.766471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb92fa3-edcd-487e-8884-528a6d94980e admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.766748 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb92fa3-edcd-487e-8884-528a6d94980e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.767016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb92fa3-edcd-487e-8884-528a6d94980e admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.767620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb92fa3-edcd-487e-8884-528a6d94980e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:27.769517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb92fa3-edcd-487e-8884-528a6d94980e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:27.841742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1228/2475] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:23:27 2022] POST /identity/v3/users => generated 293 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:27.846318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089c0d3-1674-41c3-b8d3-e2ab6a1e5e79 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:27.857151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089c0d3-1674-41c3-b8d3-e2ab6a1e5e79 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:27.862465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6c5312-e6b2-488d-b9bc-ad5f2735d3d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:27.863054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6c5312-e6b2-488d-b9bc-ad5f2735d3d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.863331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6c5312-e6b2-488d-b9bc-ad5f2735d3d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.863601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6c5312-e6b2-488d-b9bc-ad5f2735d3d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.864233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6c5312-e6b2-488d-b9bc-ad5f2735d3d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:27.865841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6c5312-e6b2-488d-b9bc-ad5f2735d3d9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:27.875929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1248/2476] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:27.879703 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb92fa3-edcd-487e-8884-528a6d94980e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:27.889663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb92fa3-edcd-487e-8884-528a6d94980e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:27.894946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd6f585-5880-4d43-8468-d9d7efee60bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:27.895558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd6f585-5880-4d43-8468-d9d7efee60bf admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.895835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd6f585-5880-4d43-8468-d9d7efee60bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.896107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd6f585-5880-4d43-8468-d9d7efee60bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.896765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd6f585-5880-4d43-8468-d9d7efee60bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:27.898407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd6f585-5880-4d43-8468-d9d7efee60bf admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:27.910043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1229/2477] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:27.913825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6c5312-e6b2-488d-b9bc-ad5f2735d3d9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:27.923723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6c5312-e6b2-488d-b9bc-ad5f2735d3d9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:27.928295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:27.928932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:27.929209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:27.929478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] PATH_INFO: `/v3/projects/640ceaf94d5445c681e5b5f29eadb0df/users/dadcc8220f9c448fb9418a311029aaf7/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:27.949609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] RBAC: Authorizing `identity:create_grant(project_id=640ceaf94d5445c681e5b5f29eadb0df, user_id=dadcc8220f9c448fb9418a311029aaf7, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:27.949609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:28.077303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1249/2478] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:23:27 2022] PUT /identity/v3/projects/640ceaf94d5445c681e5b5f29eadb0df/users/dadcc8220f9c448fb9418a311029aaf7/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:23:28.089397 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff434558-c9a3-4f8d-ab0e-b021c525bd5a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:28.089714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff434558-c9a3-4f8d-ab0e-b021c525bd5a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:28.089993 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff434558-c9a3-4f8d-ab0e-b021c525bd5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:28.117791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ff434558-c9a3-4f8d-ab0e-b021c525bd5a None None] MFA Rules not processed for user `dadcc8220f9c448fb9418a311029aaf7`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:28.118997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ff434558-c9a3-4f8d-ab0e-b021c525bd5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:28.183667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1230/2479] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:28.194966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:28.196721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:28.269969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1f7325-43a6-448a-b375-34f365f9f508 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:28.275453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3825e6-a20f-4b9f-b893-3c395cf9b22f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:28.276049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3825e6-a20f-4b9f-b893-3c395cf9b22f service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:28.276329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3825e6-a20f-4b9f-b893-3c395cf9b22f service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:28.276593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3825e6-a20f-4b9f-b893-3c395cf9b22f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:28.284276 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3825e6-a20f-4b9f-b893-3c395cf9b22f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:28.285886 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3825e6-a20f-4b9f-b893-3c395cf9b22f service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:28.294455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1250/2480] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:28 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:28.315608 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff434558-c9a3-4f8d-ab0e-b021c525bd5a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:28.317058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ff434558-c9a3-4f8d-ab0e-b021c525bd5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:28.375107 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:28.397498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff434558-c9a3-4f8d-ab0e-b021c525bd5a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:28.401379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:28.402068 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:28.402367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:28.402638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:28.404362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:28.418182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:28.419742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:28.431080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1231/2481] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:29.136703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3825e6-a20f-4b9f-b893-3c395cf9b22f service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:29.146776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3825e6-a20f-4b9f-b893-3c395cf9b22f service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:29.150581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b197fcf-a935-422f-b9d5-aa1f524bb823 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:29.151172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b197fcf-a935-422f-b9d5-aa1f524bb823 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:29.151450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b197fcf-a935-422f-b9d5-aa1f524bb823 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:29.151721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b197fcf-a935-422f-b9d5-aa1f524bb823 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:29.153401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2b197fcf-a935-422f-b9d5-aa1f524bb823 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:29.167689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b197fcf-a935-422f-b9d5-aa1f524bb823 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:29.169146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b197fcf-a935-422f-b9d5-aa1f524bb823 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:29.188519 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1251/2482] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:38.442767 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=12 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:38.857052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:38.916885 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d8c9cc-a849-4197-9c5f-817e337428bd service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:38.921420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:38.922075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:38.922356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:38.922631 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] PATH_INFO: `/v3/users/dadcc8220f9c448fb9418a311029aaf7` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:38.923749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dadcc8220f9c448fb9418a311029aaf7)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:38.926252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:39.152440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dadcc8220f9c448fb9418a311029aaf7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:39.164804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dadcc8220f9c448fb9418a311029aaf7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:39.165460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dadcc8220f9c448fb9418a311029aaf7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:39.175297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=66,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:39.270614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dadcc8220f9c448fb9418a311029aaf7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:39.278187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dadcc8220f9c448fb9418a311029aaf7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:39.284679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1232/2483] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:23:38 2022] DELETE /identity/v3/users/dadcc8220f9c448fb9418a311029aaf7 => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:39.289082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc59daf-af9b-447c-8ce7-647a2f7a77a0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:39.289379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc59daf-af9b-447c-8ce7-647a2f7a77a0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:39.289656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc59daf-af9b-447c-8ce7-647a2f7a77a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:39.308125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6cc59daf-af9b-447c-8ce7-647a2f7a77a0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:39.309306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6cc59daf-af9b-447c-8ce7-647a2f7a77a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:39.367385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1252/2484] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:39.377175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:39.378708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:39.436707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962439ff-49d9-4450-b416-13fa04f230b5 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:39.440707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a565a8f-7514-4d2e-835d-648d620ffb7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:39.441311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a565a8f-7514-4d2e-835d-648d620ffb7a service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:39.441604 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a565a8f-7514-4d2e-835d-648d620ffb7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:39.441887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a565a8f-7514-4d2e-835d-648d620ffb7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:39.450710 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a565a8f-7514-4d2e-835d-648d620ffb7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:39.452181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a565a8f-7514-4d2e-835d-648d620ffb7a service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:39.462563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1233/2485] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:39.886026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc59daf-af9b-447c-8ce7-647a2f7a77a0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:39.887705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6cc59daf-af9b-447c-8ce7-647a2f7a77a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:39.933306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc59daf-af9b-447c-8ce7-647a2f7a77a0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:39.933306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:39.933306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:39.933306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:39.934379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] PATH_INFO: `/v3/projects/640ceaf94d5445c681e5b5f29eadb0df` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:39.934379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] RBAC: Authorizing `identity:delete_project(project_id=640ceaf94d5445c681e5b5f29eadb0df)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:39.934379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:40.078056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '640ceaf94d5445c681e5b5f29eadb0df', 'request_id': 'req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:40.083547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '640ceaf94d5445c681e5b5f29eadb0df', 'request_id': 'req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:40.174608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '640ceaf94d5445c681e5b5f29eadb0df', 'request_id': 'req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:40.185860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] The token cache is being invalidate because project 640ceaf94d5445c681e5b5f29eadb0df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:23:40.186435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:40.186784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8d40925e-8e5e-492f-8e8e-91eb2bde6d09 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:40.188246 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1253/2486] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:23:39 2022] DELETE /identity/v3/projects/640ceaf94d5445c681e5b5f29eadb0df => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:23:40.194590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3dff6f-3df5-4d05-86a3-f23f4732ffc0 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.196227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3dff6f-3df5-4d05-86a3-f23f4732ffc0 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.196227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3dff6f-3df5-4d05-86a3-f23f4732ffc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.214093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1f3dff6f-3df5-4d05-86a3-f23f4732ffc0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:40.215314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1f3dff6f-3df5-4d05-86a3-f23f4732ffc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:40.232763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:40.273935 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1234/2487] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:40.282016 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529be0da-bf77-4368-a425-fd15dd01a529 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.282304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529be0da-bf77-4368-a425-fd15dd01a529 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.282580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529be0da-bf77-4368-a425-fd15dd01a529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.302883 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-529be0da-bf77-4368-a425-fd15dd01a529 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:40.304084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-529be0da-bf77-4368-a425-fd15dd01a529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:40.311918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1254/2488] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:40.316213 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3dff6f-3df5-4d05-86a3-f23f4732ffc0 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:40.325728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3dff6f-3df5-4d05-86a3-f23f4732ffc0 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:40.330676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:40.331264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.331542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.331813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.332473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:23:40.332772 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:40.334403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:40.345603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1235/2489] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:23:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:23:40.350401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529be0da-bf77-4368-a425-fd15dd01a529 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:40.359456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529be0da-bf77-4368-a425-fd15dd01a529 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:40.364617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e14e3d-6154-45dc-8a44-ee8e772ef6cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:40.365202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e14e3d-6154-45dc-8a44-ee8e772ef6cf admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.365477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e14e3d-6154-45dc-8a44-ee8e772ef6cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.365798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e14e3d-6154-45dc-8a44-ee8e772ef6cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.366409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e14e3d-6154-45dc-8a44-ee8e772ef6cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:40.368274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e14e3d-6154-45dc-8a44-ee8e772ef6cf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:40.490829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1255/2490] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:23:40 2022] POST /identity/v3/projects => generated 387 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:40.494989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:40.505208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2e4b15-d296-443e-a5fb-8818091c89f2 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:40.563172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f095c03-2e02-49a9-b743-8b7a5b91eb6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:40.563792 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f095c03-2e02-49a9-b743-8b7a5b91eb6c admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.564076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f095c03-2e02-49a9-b743-8b7a5b91eb6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.564356 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f095c03-2e02-49a9-b743-8b7a5b91eb6c admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.564981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f095c03-2e02-49a9-b743-8b7a5b91eb6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:40.567039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f095c03-2e02-49a9-b743-8b7a5b91eb6c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:40.678017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1236/2491] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:23:40 2022] POST /identity/v3/users => generated 294 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:40.681997 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e14e3d-6154-45dc-8a44-ee8e772ef6cf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:40.692088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e14e3d-6154-45dc-8a44-ee8e772ef6cf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:40.696617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7281723b-86b1-4205-a512-514fae2f8e03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:40.697203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7281723b-86b1-4205-a512-514fae2f8e03 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.697479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7281723b-86b1-4205-a512-514fae2f8e03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.697783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7281723b-86b1-4205-a512-514fae2f8e03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.698546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7281723b-86b1-4205-a512-514fae2f8e03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:40.700143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7281723b-86b1-4205-a512-514fae2f8e03 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:40.710382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1256/2492] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:40.714384 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f095c03-2e02-49a9-b743-8b7a5b91eb6c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:40.725007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f095c03-2e02-49a9-b743-8b7a5b91eb6c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:40.729564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3505523f-88c1-4ad5-80d2-626d882675bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:40.730178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505523f-88c1-4ad5-80d2-626d882675bb admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.730458 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505523f-88c1-4ad5-80d2-626d882675bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.730732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505523f-88c1-4ad5-80d2-626d882675bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.731363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3505523f-88c1-4ad5-80d2-626d882675bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:40.732947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3505523f-88c1-4ad5-80d2-626d882675bb admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:40.743209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1237/2493] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:40.746988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7281723b-86b1-4205-a512-514fae2f8e03 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:40.756560 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7281723b-86b1-4205-a512-514fae2f8e03 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:40.761304 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc1d54-9cf5-43bd-a6bf-63bb46c6767a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:40.761979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc1d54-9cf5-43bd-a6bf-63bb46c6767a admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.762256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc1d54-9cf5-43bd-a6bf-63bb46c6767a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.762528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fc1d54-9cf5-43bd-a6bf-63bb46c6767a admin admin] PATH_INFO: `/v3/projects/b0090a1572f442fc8dc3df388aee6b2b/users/b195072164ee4189947ec273ed64bd75/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.777259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fc1d54-9cf5-43bd-a6bf-63bb46c6767a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0090a1572f442fc8dc3df388aee6b2b, user_id=b195072164ee4189947ec273ed64bd75, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:40.779930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fc1d54-9cf5-43bd-a6bf-63bb46c6767a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:40.848832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1257/2494] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:23:40 2022] PUT /identity/v3/projects/b0090a1572f442fc8dc3df388aee6b2b/users/b195072164ee4189947ec273ed64bd75/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:40.859025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9907a-7958-442a-8c4c-a667ccff6796 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:40.859304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9907a-7958-442a-8c4c-a667ccff6796 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:40.859596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9907a-7958-442a-8c4c-a667ccff6796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:40.886380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-98e9907a-7958-442a-8c4c-a667ccff6796 None None] MFA Rules not processed for user `b195072164ee4189947ec273ed64bd75`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:40.887587 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-98e9907a-7958-442a-8c4c-a667ccff6796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:40.950881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1238/2495] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:40 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:40.956024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc1d54-9cf5-43bd-a6bf-63bb46c6767a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:40.977408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:41.017123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fc1d54-9cf5-43bd-a6bf-63bb46c6767a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:41.022399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7bcf50-6655-4904-8e68-b590c64fd780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:41.023023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7bcf50-6655-4904-8e68-b590c64fd780 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:41.023302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7bcf50-6655-4904-8e68-b590c64fd780 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:41.023578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7bcf50-6655-4904-8e68-b590c64fd780 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:41.024231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7bcf50-6655-4904-8e68-b590c64fd780 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:41.026169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7bcf50-6655-4904-8e68-b590c64fd780 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:41.169080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1258/2496] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:23:40 2022] POST /identity/v3/projects => generated 385 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:41.173166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e9907a-7958-442a-8c4c-a667ccff6796 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:41.183875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e9907a-7958-442a-8c4c-a667ccff6796 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:41.196313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:41.240110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621694b0-54a2-486f-aa43-913feb49c91d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:41.240807 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621694b0-54a2-486f-aa43-913feb49c91d admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:41.241085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621694b0-54a2-486f-aa43-913feb49c91d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:41.241378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621694b0-54a2-486f-aa43-913feb49c91d admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:41.242052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621694b0-54a2-486f-aa43-913feb49c91d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:41.243957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621694b0-54a2-486f-aa43-913feb49c91d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:41.397764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1239/2497] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:23:41 2022] POST /identity/v3/users => generated 299 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:41.401765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7bcf50-6655-4904-8e68-b590c64fd780 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:41.412064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7bcf50-6655-4904-8e68-b590c64fd780 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:41.416743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec1a640-89a8-459a-89c3-9cac3c226a8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:41.417345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec1a640-89a8-459a-89c3-9cac3c226a8e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:41.417662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec1a640-89a8-459a-89c3-9cac3c226a8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:41.417944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec1a640-89a8-459a-89c3-9cac3c226a8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:41.418613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec1a640-89a8-459a-89c3-9cac3c226a8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:41.420199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec1a640-89a8-459a-89c3-9cac3c226a8e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:41.432272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1259/2498] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:41.436394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621694b0-54a2-486f-aa43-913feb49c91d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:41.447066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621694b0-54a2-486f-aa43-913feb49c91d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:41.451678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb72f93-bc0e-45f3-9330-9863e911e763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:41.452328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb72f93-bc0e-45f3-9330-9863e911e763 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:41.452610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb72f93-bc0e-45f3-9330-9863e911e763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:41.452887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb72f93-bc0e-45f3-9330-9863e911e763 admin admin] PATH_INFO: `/v3/projects/20bf1e0509024d98b6bb596b2dfb88aa/users/f7c3ff356f4f4ca38ce0aead17115eee/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:41.468740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb72f93-bc0e-45f3-9330-9863e911e763 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20bf1e0509024d98b6bb596b2dfb88aa, user_id=f7c3ff356f4f4ca38ce0aead17115eee, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:41.471429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb72f93-bc0e-45f3-9330-9863e911e763 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:41.579546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1240/2499] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:23:41 2022] PUT /identity/v3/projects/20bf1e0509024d98b6bb596b2dfb88aa/users/f7c3ff356f4f4ca38ce0aead17115eee/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:41.583481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec1a640-89a8-459a-89c3-9cac3c226a8e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:41.593288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec1a640-89a8-459a-89c3-9cac3c226a8e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:41.651184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6f1295-6186-4f25-b15f-fb577c9da100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:41.651724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f1295-6186-4f25-b15f-fb577c9da100 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:41.652000 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f1295-6186-4f25-b15f-fb577c9da100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:41.652271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6f1295-6186-4f25-b15f-fb577c9da100 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:41.652906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6f1295-6186-4f25-b15f-fb577c9da100 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:41.654521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6f1295-6186-4f25-b15f-fb577c9da100 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:41.664019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1260/2500] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:41 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:41.669804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb72f93-bc0e-45f3-9330-9863e911e763 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:41.680382 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb72f93-bc0e-45f3-9330-9863e911e763 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:41.684898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:41.685539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:41.685848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:41.686113 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] PATH_INFO: `/v3/projects/20bf1e0509024d98b6bb596b2dfb88aa/users/f7c3ff356f4f4ca38ce0aead17115eee/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:41.687998 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20bf1e0509024d98b6bb596b2dfb88aa, user_id=f7c3ff356f4f4ca38ce0aead17115eee, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:41.691540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:41.701377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1241/2501] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:23:41 2022] PUT /identity/v3/projects/20bf1e0509024d98b6bb596b2dfb88aa/users/f7c3ff356f4f4ca38ce0aead17115eee/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:41.711295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eba2ebf-f542-4c44-94ac-a03118e5223d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:41.711707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eba2ebf-f542-4c44-94ac-a03118e5223d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:41.711987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eba2ebf-f542-4c44-94ac-a03118e5223d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:41.740266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-8eba2ebf-f542-4c44-94ac-a03118e5223d None None] MFA Rules not processed for user `f7c3ff356f4f4ca38ce0aead17115eee`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:41.741455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8eba2ebf-f542-4c44-94ac-a03118e5223d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:41.807199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1261/2502] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:41 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:23:41.818476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:41.820158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:41.876257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8c6a04-735b-44ce-958c-5af81b059157 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:41.879851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccb20dc-9bf4-4cbb-8ea6-d8961b62ff12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:41.880442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb20dc-9bf4-4cbb-8ea6-d8961b62ff12 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:41.880715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb20dc-9bf4-4cbb-8ea6-d8961b62ff12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:41.880981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccb20dc-9bf4-4cbb-8ea6-d8961b62ff12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:41.889791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccb20dc-9bf4-4cbb-8ea6-d8961b62ff12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:41.891248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccb20dc-9bf4-4cbb-8ea6-d8961b62ff12 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:41.901965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1242/2503] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:41 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:41.969506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eba2ebf-f542-4c44-94ac-a03118e5223d None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:41.988877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:42.032621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eba2ebf-f542-4c44-94ac-a03118e5223d None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:42.038017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:42.038641 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:42.038920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:42.039193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] PATH_INFO: `/v3/users/b195072164ee4189947ec273ed64bd75` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:42.040315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] RBAC: Authorizing `identity:delete_user(user_id=b195072164ee4189947ec273ed64bd75)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:42.042263 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:42.170153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b195072164ee4189947ec273ed64bd75'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:42.170683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b195072164ee4189947ec273ed64bd75'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:42.182484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b195072164ee4189947ec273ed64bd75'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:42.271082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b195072164ee4189947ec273ed64bd75'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:42.278378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b195072164ee4189947ec273ed64bd75'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:42.284803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1262/2504] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:23:41 2022] DELETE /identity/v3/users/b195072164ee4189947ec273ed64bd75 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:42.289338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312e54a0-5784-420e-95b0-b3fcf36f723e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:42.289640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312e54a0-5784-420e-95b0-b3fcf36f723e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:42.289913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312e54a0-5784-420e-95b0-b3fcf36f723e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:42.308420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-312e54a0-5784-420e-95b0-b3fcf36f723e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:42.309620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-312e54a0-5784-420e-95b0-b3fcf36f723e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:42.370063 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1243/2505] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:42.380175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:42.381940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:42.438632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696bdf3-9537-449b-9a3a-b276f8112bdd admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:42.442726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5dca17-4a45-4eaa-9469-0e94c6b44cc4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:42.443333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5dca17-4a45-4eaa-9469-0e94c6b44cc4 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:42.443611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5dca17-4a45-4eaa-9469-0e94c6b44cc4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:42.443914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5dca17-4a45-4eaa-9469-0e94c6b44cc4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:42.452608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5dca17-4a45-4eaa-9469-0e94c6b44cc4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:42.454069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5dca17-4a45-4eaa-9469-0e94c6b44cc4 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:42.463799 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1263/2506] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:42.844116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312e54a0-5784-420e-95b0-b3fcf36f723e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:42.845768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-312e54a0-5784-420e-95b0-b3fcf36f723e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:42.857844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312e54a0-5784-420e-95b0-b3fcf36f723e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:42.863211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:42.863849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:42.864122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:42.864388 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] PATH_INFO: `/v3/projects/b0090a1572f442fc8dc3df388aee6b2b` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:42.865508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0090a1572f442fc8dc3df388aee6b2b)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:42.867449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:42.877481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:43.008727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0090a1572f442fc8dc3df388aee6b2b', 'request_id': 'req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.093912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0090a1572f442fc8dc3df388aee6b2b', 'request_id': 'req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.100132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0090a1572f442fc8dc3df388aee6b2b', 'request_id': 'req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.112275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] The token cache is being invalidate because project b0090a1572f442fc8dc3df388aee6b2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:23:43.112559 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.113186 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.114672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1244/2507] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:23:42 2022] DELETE /identity/v3/projects/b0090a1572f442fc8dc3df388aee6b2b => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:43.118634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5dca17-4a45-4eaa-9469-0e94c6b44cc4 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:43.128830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5dca17-4a45-4eaa-9469-0e94c6b44cc4 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:43.160982 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:43.183475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:43.184111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:43.184393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:43.184668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] PATH_INFO: `/v3/users/f7c3ff356f4f4ca38ce0aead17115eee` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:43.185827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7c3ff356f4f4ca38ce0aead17115eee)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:43.187817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:43.359085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7c3ff356f4f4ca38ce0aead17115eee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.359666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7c3ff356f4f4ca38ce0aead17115eee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.372311 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7c3ff356f4f4ca38ce0aead17115eee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.448488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7c3ff356f4f4ca38ce0aead17115eee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.456028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d28f7460-33f9-4e2d-9529-ef3d95128703 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7c3ff356f4f4ca38ce0aead17115eee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:43.461912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1264/2508] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:23:43 2022] DELETE /identity/v3/users/f7c3ff356f4f4ca38ce0aead17115eee => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:43.896596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:43.898234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:43.912174 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2b6fa3-293a-4785-ad6d-75b2ca37677d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:43.969367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:43.970055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:43.970332 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:43.970602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] PATH_INFO: `/v3/projects/20bf1e0509024d98b6bb596b2dfb88aa` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:43.971706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] RBAC: Authorizing `identity:delete_project(project_id=20bf1e0509024d98b6bb596b2dfb88aa)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:43.973949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:44.128419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20bf1e0509024d98b6bb596b2dfb88aa', 'request_id': 'req-f0ef6878-8516-4f50-bd39-94a526ac0077'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:44.225644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20bf1e0509024d98b6bb596b2dfb88aa', 'request_id': 'req-f0ef6878-8516-4f50-bd39-94a526ac0077'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:44.231735 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20bf1e0509024d98b6bb596b2dfb88aa', 'request_id': 'req-f0ef6878-8516-4f50-bd39-94a526ac0077'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:44.242904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] The token cache is being invalidate because project 20bf1e0509024d98b6bb596b2dfb88aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:23:44.243191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:44.243829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-f0ef6878-8516-4f50-bd39-94a526ac0077 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:44.245521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1245/2509] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:23:43 2022] DELETE /identity/v3/projects/20bf1e0509024d98b6bb596b2dfb88aa => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:44.250183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c1fe1-5169-492a-b81c-19424c8e05b8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.250478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c1fe1-5169-492a-b81c-19424c8e05b8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.250753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c1fe1-5169-492a-b81c-19424c8e05b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.271290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-131c1fe1-5169-492a-b81c-19424c8e05b8 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:44.276269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-131c1fe1-5169-492a-b81c-19424c8e05b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:44.329465 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1265/2510] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:44.337481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82393d5-b5f0-4d18-8f4c-abfc2a23fc64 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.338969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82393d5-b5f0-4d18-8f4c-abfc2a23fc64 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.338969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82393d5-b5f0-4d18-8f4c-abfc2a23fc64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.359296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-a82393d5-b5f0-4d18-8f4c-abfc2a23fc64 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:44.360554 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a82393d5-b5f0-4d18-8f4c-abfc2a23fc64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:44.369173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1246/2511] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:44.373351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131c1fe1-5169-492a-b81c-19424c8e05b8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:44.383686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131c1fe1-5169-492a-b81c-19424c8e05b8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:44.388327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:44.390336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.390336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.390336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.390336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:23:44.390748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:44.392018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:44.403650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1266/2512] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:23:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:23:44.407959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82393d5-b5f0-4d18-8f4c-abfc2a23fc64 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:44.417042 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82393d5-b5f0-4d18-8f4c-abfc2a23fc64 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:44.421573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b153b9-8e61-4b55-91fc-cfed3b4eda90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:44.422185 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b153b9-8e61-4b55-91fc-cfed3b4eda90 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.422463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b153b9-8e61-4b55-91fc-cfed3b4eda90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.422726 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b153b9-8e61-4b55-91fc-cfed3b4eda90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.423342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b153b9-8e61-4b55-91fc-cfed3b4eda90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:44.425788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b153b9-8e61-4b55-91fc-cfed3b4eda90 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:44.529016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:44.536089 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1247/2513] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:23:44 2022] POST /identity/v3/projects => generated 373 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:44.540321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:44.550899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e43fdd-e894-4bab-8cb2-1ed37fe77f43 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:44.593213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:44.607904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297c711e-164f-4b5f-b8bc-d6519a9137ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:44.608505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297c711e-164f-4b5f-b8bc-d6519a9137ef admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.608785 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297c711e-164f-4b5f-b8bc-d6519a9137ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.609058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297c711e-164f-4b5f-b8bc-d6519a9137ef admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.609723 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297c711e-164f-4b5f-b8bc-d6519a9137ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:44.612132 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297c711e-164f-4b5f-b8bc-d6519a9137ef admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:44.757643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1267/2514] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:23:44 2022] POST /identity/v3/users => generated 287 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:44.762060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b153b9-8e61-4b55-91fc-cfed3b4eda90 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:44.772445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b153b9-8e61-4b55-91fc-cfed3b4eda90 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:44.778196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b923dd35-5ac4-44cd-9674-d76368bfdaa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:44.778800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923dd35-5ac4-44cd-9674-d76368bfdaa7 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.779129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923dd35-5ac4-44cd-9674-d76368bfdaa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.779373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b923dd35-5ac4-44cd-9674-d76368bfdaa7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.780387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b923dd35-5ac4-44cd-9674-d76368bfdaa7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:44.782011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b923dd35-5ac4-44cd-9674-d76368bfdaa7 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:44.796546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1248/2515] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:44.800010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297c711e-164f-4b5f-b8bc-d6519a9137ef admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:44.817061 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297c711e-164f-4b5f-b8bc-d6519a9137ef admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:44.822035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8799c190-4fc5-4fd0-bfae-36fe03283ac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:44.822638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799c190-4fc5-4fd0-bfae-36fe03283ac2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.822914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799c190-4fc5-4fd0-bfae-36fe03283ac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.823185 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8799c190-4fc5-4fd0-bfae-36fe03283ac2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.823828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8799c190-4fc5-4fd0-bfae-36fe03283ac2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:44.825517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8799c190-4fc5-4fd0-bfae-36fe03283ac2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:44.837038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1268/2516] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:44 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:44.843009 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b923dd35-5ac4-44cd-9674-d76368bfdaa7 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:44.853882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b923dd35-5ac4-44cd-9674-d76368bfdaa7 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:44.858456 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:44.859043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.859324 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.859596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] PATH_INFO: `/v3/projects/288f8f920be04985bc3ba5706cf20ecf/users/3d9a18c3db1f42e7a889ea658affb711/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.874373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] RBAC: Authorizing `identity:create_grant(project_id=288f8f920be04985bc3ba5706cf20ecf, user_id=3d9a18c3db1f42e7a889ea658affb711, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:44.877088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:44.957247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1249/2517] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:23:44 2022] PUT /identity/v3/projects/288f8f920be04985bc3ba5706cf20ecf/users/3d9a18c3db1f42e7a889ea658affb711/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:44.968824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcccbfd8-4ff7-42f0-99af-3a728e84f8c9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:44.969090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcccbfd8-4ff7-42f0-99af-3a728e84f8c9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:44.969359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcccbfd8-4ff7-42f0-99af-3a728e84f8c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:44.998303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bcccbfd8-4ff7-42f0-99af-3a728e84f8c9 None None] MFA Rules not processed for user `3d9a18c3db1f42e7a889ea658affb711`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:44.999625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bcccbfd8-4ff7-42f0-99af-3a728e84f8c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:45.063633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1269/2518] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:45.074565 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:45.076292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:45.134286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f27c68-2825-4bf5-8cc8-532546b4e720 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:45.137936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5931295a-f5c5-4ba5-94bb-8ea2cf5693d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:45.138535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5931295a-f5c5-4ba5-94bb-8ea2cf5693d3 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:45.138810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5931295a-f5c5-4ba5-94bb-8ea2cf5693d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:45.139082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5931295a-f5c5-4ba5-94bb-8ea2cf5693d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:45.147695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5931295a-f5c5-4ba5-94bb-8ea2cf5693d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:45.149108 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5931295a-f5c5-4ba5-94bb-8ea2cf5693d3 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:45.158016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1250/2519] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:45.179539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcccbfd8-4ff7-42f0-99af-3a728e84f8c9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:45.246942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcccbfd8-4ff7-42f0-99af-3a728e84f8c9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:45.251546 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:45.253367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:45.253367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:45.253367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] PATH_INFO: `/v3/users/3d9a18c3db1f42e7a889ea658affb711` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:45.253857 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d9a18c3db1f42e7a889ea658affb711)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:45.255768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:45.485702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d9a18c3db1f42e7a889ea658affb711'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:45.486354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d9a18c3db1f42e7a889ea658affb711'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:45.491525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:45.498329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d9a18c3db1f42e7a889ea658affb711'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:45.591065 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d9a18c3db1f42e7a889ea658affb711'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:45.598691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d9a18c3db1f42e7a889ea658affb711'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:45.604578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1270/2520] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:23:45 2022] DELETE /identity/v3/users/3d9a18c3db1f42e7a889ea658affb711 => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:45.609166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13901bed-dcfc-4cdf-8193-678907c3246e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:45.609450 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13901bed-dcfc-4cdf-8193-678907c3246e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:45.609750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13901bed-dcfc-4cdf-8193-678907c3246e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:45.628622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-13901bed-dcfc-4cdf-8193-678907c3246e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:45.629844 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-13901bed-dcfc-4cdf-8193-678907c3246e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:45.686169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1251/2521] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:45.696258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:45.698077 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:45.754446 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783a75fb-3fca-4702-be5f-7593b1b67f56 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:45.757893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51115502-7656-43d2-947e-7cc7954e02b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:45.758485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51115502-7656-43d2-947e-7cc7954e02b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:45.758762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51115502-7656-43d2-947e-7cc7954e02b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:45.759291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51115502-7656-43d2-947e-7cc7954e02b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:45.769047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51115502-7656-43d2-947e-7cc7954e02b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:45.770515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51115502-7656-43d2-947e-7cc7954e02b4 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:45.779802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1271/2522] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:46.239336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13901bed-dcfc-4cdf-8193-678907c3246e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:46.241530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-13901bed-dcfc-4cdf-8193-678907c3246e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:46.252964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13901bed-dcfc-4cdf-8193-678907c3246e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:46.258005 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:46.258532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:46.258811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:46.259087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] PATH_INFO: `/v3/projects/288f8f920be04985bc3ba5706cf20ecf` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:46.260285 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] RBAC: Authorizing `identity:delete_project(project_id=288f8f920be04985bc3ba5706cf20ecf)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:46.270590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:46.421161 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '288f8f920be04985bc3ba5706cf20ecf', 'request_id': 'req-6f17f4e0-2e9a-45fd-881a-9adf193a6078'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:46.563195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '288f8f920be04985bc3ba5706cf20ecf', 'request_id': 'req-6f17f4e0-2e9a-45fd-881a-9adf193a6078'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:46.567636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:46.573233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '288f8f920be04985bc3ba5706cf20ecf', 'request_id': 'req-6f17f4e0-2e9a-45fd-881a-9adf193a6078'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:46.581665 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] The token cache is being invalidate because project 288f8f920be04985bc3ba5706cf20ecf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:23:46.581957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:46.582594 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6f17f4e0-2e9a-45fd-881a-9adf193a6078 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:23:46.584048 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1252/2523] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:23:46 2022] DELETE /identity/v3/projects/288f8f920be04985bc3ba5706cf20ecf => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:46.589841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf62bc2-2719-4649-a946-5c0d6b16d210 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:46.590136 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf62bc2-2719-4649-a946-5c0d6b16d210 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:46.590430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf62bc2-2719-4649-a946-5c0d6b16d210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:46.610259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ccf62bc2-2719-4649-a946-5c0d6b16d210 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:46.611499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ccf62bc2-2719-4649-a946-5c0d6b16d210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:46.670608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1272/2524] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:46.678529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da3861b-641d-4c5c-b5cf-42f51d58ab8e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:46.678818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da3861b-641d-4c5c-b5cf-42f51d58ab8e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:46.679096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da3861b-641d-4c5c-b5cf-42f51d58ab8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:46.698087 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-3da3861b-641d-4c5c-b5cf-42f51d58ab8e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:46.699377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3da3861b-641d-4c5c-b5cf-42f51d58ab8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:46.707708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1253/2525] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:46.711882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf62bc2-2719-4649-a946-5c0d6b16d210 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:46.722601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf62bc2-2719-4649-a946-5c0d6b16d210 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:46.727507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:46.728102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:46.728488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:46.728762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:46.729403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:23:46.729743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:46.731382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:46.742738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1273/2526] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:23:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:23:46.746930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da3861b-641d-4c5c-b5cf-42f51d58ab8e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:46.756106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da3861b-641d-4c5c-b5cf-42f51d58ab8e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:46.760707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c1802e-b747-4bd0-b937-ed151f727e6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:46.761298 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c1802e-b747-4bd0-b937-ed151f727e6d admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:46.761574 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c1802e-b747-4bd0-b937-ed151f727e6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:46.761878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c1802e-b747-4bd0-b937-ed151f727e6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:46.766373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c1802e-b747-4bd0-b937-ed151f727e6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:46.766373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c1802e-b747-4bd0-b937-ed151f727e6d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:46.855144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1254/2527] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:23:46 2022] POST /identity/v3/projects => generated 383 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:46.859335 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:46.869225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db7616a-4165-46f9-b859-f8c95fa187e2 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:46.925644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ce6494-612e-4fa5-98ae-db6bc6f7f3d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:46.926251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ce6494-612e-4fa5-98ae-db6bc6f7f3d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:46.926528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ce6494-612e-4fa5-98ae-db6bc6f7f3d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:46.927084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ce6494-612e-4fa5-98ae-db6bc6f7f3d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:46.927702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ce6494-612e-4fa5-98ae-db6bc6f7f3d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:46.929621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ce6494-612e-4fa5-98ae-db6bc6f7f3d3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:46.938343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:23:47.020325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1274/2528] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:23:46 2022] POST /identity/v3/users => generated 292 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:23:47.024548 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c1802e-b747-4bd0-b937-ed151f727e6d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:47.034917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c1802e-b747-4bd0-b937-ed151f727e6d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:47.040281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53463dd-1b01-4535-9aa8-c20cd4f03ad2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:47.040918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53463dd-1b01-4535-9aa8-c20cd4f03ad2 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:47.041194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53463dd-1b01-4535-9aa8-c20cd4f03ad2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:47.041462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53463dd-1b01-4535-9aa8-c20cd4f03ad2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:47.042164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53463dd-1b01-4535-9aa8-c20cd4f03ad2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:47.043793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53463dd-1b01-4535-9aa8-c20cd4f03ad2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:47.053979 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1255/2529] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:23:47.057887 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ce6494-612e-4fa5-98ae-db6bc6f7f3d3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:47.076198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ce6494-612e-4fa5-98ae-db6bc6f7f3d3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:47.085218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89bdc02-d360-44d1-b252-23126b3ac253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:47.086006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89bdc02-d360-44d1-b252-23126b3ac253 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:47.086358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89bdc02-d360-44d1-b252-23126b3ac253 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:47.086714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89bdc02-d360-44d1-b252-23126b3ac253 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:47.087440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89bdc02-d360-44d1-b252-23126b3ac253 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:47.089158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89bdc02-d360-44d1-b252-23126b3ac253 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:47.102072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1275/2530] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:23:47 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:23:47.106542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53463dd-1b01-4535-9aa8-c20cd4f03ad2 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:47.116153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53463dd-1b01-4535-9aa8-c20cd4f03ad2 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:47.120819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:47.121493 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:47.121801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:47.122079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] PATH_INFO: `/v3/projects/aed82d3b5299415ca1e2e3cfcb7e2d0b/users/6a4136b08cc1464eac6c779652a1d2b1/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:47.137255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] RBAC: Authorizing `identity:create_grant(project_id=aed82d3b5299415ca1e2e3cfcb7e2d0b, user_id=6a4136b08cc1464eac6c779652a1d2b1, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:47.139983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:47.214073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1256/2531] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:23:47 2022] PUT /identity/v3/projects/aed82d3b5299415ca1e2e3cfcb7e2d0b/users/6a4136b08cc1464eac6c779652a1d2b1/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:23:47.223980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab517ab-afe8-41db-826f-14fc83a448b2 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:47.224441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab517ab-afe8-41db-826f-14fc83a448b2 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:47.224894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab517ab-afe8-41db-826f-14fc83a448b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:47.244161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3ab517ab-afe8-41db-826f-14fc83a448b2 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:47.246317 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3ab517ab-afe8-41db-826f-14fc83a448b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:47.309482 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1276/2532] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:47.319362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:47.321217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:47.377432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5b800a-109e-48df-9030-72d3f63d1bae admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:47.381052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:47.381681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:47.381962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:47.382229 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:47.391158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:47.393119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:47.404417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1257/2533] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:48.918195 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0da759-a7bd-4a37-ba07-fc0936d9ef66 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:48.918773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0da759-a7bd-4a37-ba07-fc0936d9ef66 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:48.918948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0da759-a7bd-4a37-ba07-fc0936d9ef66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:48.948607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0d0da759-a7bd-4a37-ba07-fc0936d9ef66 None None] MFA Rules not processed for user `6a4136b08cc1464eac6c779652a1d2b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:48.949941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0d0da759-a7bd-4a37-ba07-fc0936d9ef66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:49.025536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1277/2534] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:49.037279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:49.040481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:49.098742 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd3db02-2200-40af-bdea-3a0fda097fea service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:49.102366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:49.102966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:49.103241 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:49.103510 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:49.112031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:49.113477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:49.124220 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1258/2535] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:23:51.926154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62039943-ea58-4099-8d7f-731a565e76b9 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:51.926705 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62039943-ea58-4099-8d7f-731a565e76b9 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:51.926825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62039943-ea58-4099-8d7f-731a565e76b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:51.951487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-62039943-ea58-4099-8d7f-731a565e76b9 None None] MFA Rules not processed for user `6a4136b08cc1464eac6c779652a1d2b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:23:51.952719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-62039943-ea58-4099-8d7f-731a565e76b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:51.972745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1278/2536] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:23:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:23:59.063697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:23:59.065162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:59.076973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3e19a3-9819-461d-a1a9-7d622d68e4a0 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:23:59.080471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:23:59.081062 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:23:59.081342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:23:59.081643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:23:59.083280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:23:59.094588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:23:59.096007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:23:59.106176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1259/2537] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:23:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:01.948511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:INSERT=2,keystone:SELECT=69 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:03.268487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62039943-ea58-4099-8d7f-731a565e76b9 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:03.270690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-62039943-ea58-4099-8d7f-731a565e76b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:03.368569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62039943-ea58-4099-8d7f-731a565e76b9 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:03.400614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:03.400614 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:03.401207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:03.401207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:03.401207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:03.401207 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:03.405610 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:03.423802 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ac94b7-ee23-4f2d-8de9-3fe941fa9c45 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:03.433467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:03.434469 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:03.435441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:03.436277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:03.436789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:03.438150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] PATH_INFO: `/v3/users/ec39d995899c4d43bb4f8ffd9e6d2ef3` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:03.448677 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1279/2538] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:03.470171 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec39d995899c4d43bb4f8ffd9e6d2ef3)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:03.473169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:03.474946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=1,keystone:SELECT=97,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:03.702289 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec39d995899c4d43bb4f8ffd9e6d2ef3'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:03.714938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec39d995899c4d43bb4f8ffd9e6d2ef3'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:03.715626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec39d995899c4d43bb4f8ffd9e6d2ef3'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:03.797223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec39d995899c4d43bb4f8ffd9e6d2ef3'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:03.804944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-19c4f179-0b4b-473c-a788-023387133039 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec39d995899c4d43bb4f8ffd9e6d2ef3'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:03.810839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1260/2539] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:03 2022] DELETE /identity/v3/users/ec39d995899c4d43bb4f8ffd9e6d2ef3 => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:04.131845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:04.147692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a79a179-a008-4809-8172-6a98e65cc990 service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:04.227683 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:04.228408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:04.228695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:04.229073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] PATH_INFO: `/v3/projects/f640cb592ed1455a827976752ce2f49d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:04.242665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=f640cb592ed1455a827976752ce2f49d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:04.245527 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:04.398574 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f640cb592ed1455a827976752ce2f49d', 'request_id': 'req-d7617438-49b3-45cc-a70c-fbc19e94d7dc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:04.404266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f640cb592ed1455a827976752ce2f49d', 'request_id': 'req-d7617438-49b3-45cc-a70c-fbc19e94d7dc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:04.523536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f640cb592ed1455a827976752ce2f49d', 'request_id': 'req-d7617438-49b3-45cc-a70c-fbc19e94d7dc'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:04.537279 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] The token cache is being invalidate because project f640cb592ed1455a827976752ce2f49d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:04.537566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:04.538228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d7617438-49b3-45cc-a70c-fbc19e94d7dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:04.539932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1280/2540] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:04 2022] DELETE /identity/v3/projects/f640cb592ed1455a827976752ce2f49d => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:04.544560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1d01e1-bccb-4e01-8920-041448e3d659 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:04.545052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1d01e1-bccb-4e01-8920-041448e3d659 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:04.545518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1d01e1-bccb-4e01-8920-041448e3d659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:04.564899 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-da1d01e1-bccb-4e01-8920-041448e3d659 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:04.566255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-da1d01e1-bccb-4e01-8920-041448e3d659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:04.637953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1261/2541] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:04.646068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a51512-9306-4bd7-9da4-bd03401cc099 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:04.646382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a51512-9306-4bd7-9da4-bd03401cc099 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:04.646790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a51512-9306-4bd7-9da4-bd03401cc099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:04.669457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e8a51512-9306-4bd7-9da4-bd03401cc099 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:04.670806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e8a51512-9306-4bd7-9da4-bd03401cc099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:04.679266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1281/2542] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:24:04.689369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1d01e1-bccb-4e01-8920-041448e3d659 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:04.700344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1d01e1-bccb-4e01-8920-041448e3d659 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:04.705144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:04.706308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:04.706593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:04.706868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:04.707522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:24:04.707826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:04.722224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:04.723854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1262/2543] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:24:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:24:04.729639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a51512-9306-4bd7-9da4-bd03401cc099 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:04.735364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:04.739319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a51512-9306-4bd7-9da4-bd03401cc099 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:04.744324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755cd6e-e16a-481d-98ab-98579d6aea9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:04.744937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755cd6e-e16a-481d-98ab-98579d6aea9c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:04.745223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755cd6e-e16a-481d-98ab-98579d6aea9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:04.745496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755cd6e-e16a-481d-98ab-98579d6aea9c admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:04.746173 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1755cd6e-e16a-481d-98ab-98579d6aea9c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:04.748080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1755cd6e-e16a-481d-98ab-98579d6aea9c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:04.873131 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1282/2544] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:04 2022] POST /identity/v3/projects => generated 357 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:04.877252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:04.887227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de6549b-aafe-4580-90cc-97b916e75973 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:04.943409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b64f13-d192-4e39-8137-b340636d736a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:04.944012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b64f13-d192-4e39-8137-b340636d736a admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:04.944390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b64f13-d192-4e39-8137-b340636d736a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:04.944667 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b64f13-d192-4e39-8137-b340636d736a admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:04.945291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b64f13-d192-4e39-8137-b340636d736a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:04.947236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b64f13-d192-4e39-8137-b340636d736a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:04.958449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:05.067955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1263/2545] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:04 2022] POST /identity/v3/users => generated 279 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:24:05.072151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755cd6e-e16a-481d-98ab-98579d6aea9c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:05.083695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755cd6e-e16a-481d-98ab-98579d6aea9c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:05.088680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261eab8d-f639-4d61-ba00-7be5061ff213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:05.089273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261eab8d-f639-4d61-ba00-7be5061ff213 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:05.089549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261eab8d-f639-4d61-ba00-7be5061ff213 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:05.089871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261eab8d-f639-4d61-ba00-7be5061ff213 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:05.090507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261eab8d-f639-4d61-ba00-7be5061ff213 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:05.092129 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261eab8d-f639-4d61-ba00-7be5061ff213 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:05.102386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1283/2546] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:05.106486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b64f13-d192-4e39-8137-b340636d736a admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:05.116476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b64f13-d192-4e39-8137-b340636d736a admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:05.121150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac91ce6-3e2f-4a5e-afca-e010a0c127a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:05.121815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac91ce6-3e2f-4a5e-afca-e010a0c127a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:05.122098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac91ce6-3e2f-4a5e-afca-e010a0c127a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:05.122386 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac91ce6-3e2f-4a5e-afca-e010a0c127a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:05.123006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac91ce6-3e2f-4a5e-afca-e010a0c127a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:05.124664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac91ce6-3e2f-4a5e-afca-e010a0c127a6 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:05.135550 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1264/2547] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:05.139565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261eab8d-f639-4d61-ba00-7be5061ff213 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:05.149868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261eab8d-f639-4d61-ba00-7be5061ff213 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:05.156665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0925c4aa-05a3-4397-b0c6-23275d91837b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:05.165718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0925c4aa-05a3-4397-b0c6-23275d91837b admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:05.165718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0925c4aa-05a3-4397-b0c6-23275d91837b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:05.165718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0925c4aa-05a3-4397-b0c6-23275d91837b admin admin] PATH_INFO: `/v3/projects/5bfd5106918c494a962cb615d1c20b87/users/5a1c3b4228e84dd7b5a5157368264a5f/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:05.182450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0925c4aa-05a3-4397-b0c6-23275d91837b admin admin] RBAC: Authorizing `identity:create_grant(project_id=5bfd5106918c494a962cb615d1c20b87, user_id=5a1c3b4228e84dd7b5a5157368264a5f, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:05.185377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0925c4aa-05a3-4397-b0c6-23275d91837b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:05.277975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1284/2548] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:05 2022] PUT /identity/v3/projects/5bfd5106918c494a962cb615d1c20b87/users/5a1c3b4228e84dd7b5a5157368264a5f/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:05.288341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd768e1-81ca-42b6-b443-210c1c8db08a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:05.288675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd768e1-81ca-42b6-b443-210c1c8db08a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:05.288952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd768e1-81ca-42b6-b443-210c1c8db08a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:05.316769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-bdd768e1-81ca-42b6-b443-210c1c8db08a None None] MFA Rules not processed for user `5a1c3b4228e84dd7b5a5157368264a5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:05.318025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-bdd768e1-81ca-42b6-b443-210c1c8db08a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:05.379895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1265/2549] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:05 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:24:05.384297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0925c4aa-05a3-4397-b0c6-23275d91837b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:05.466803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0925c4aa-05a3-4397-b0c6-23275d91837b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:05.471504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8a88c1-9938-4d32-afc0-9019f526c4a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:05.472133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8a88c1-9938-4d32-afc0-9019f526c4a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:05.472406 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8a88c1-9938-4d32-afc0-9019f526c4a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:05.472679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8a88c1-9938-4d32-afc0-9019f526c4a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:05.473972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8a88c1-9938-4d32-afc0-9019f526c4a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:05.475951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8a88c1-9938-4d32-afc0-9019f526c4a3 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:05.641473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1285/2550] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:05 2022] POST /identity/v3/projects => generated 357 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:05.642883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd768e1-81ca-42b6-b443-210c1c8db08a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:05.653751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd768e1-81ca-42b6-b443-210c1c8db08a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:05.710464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a3e121-09d2-4f97-90ec-616c3d489f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:05.711088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3e121-09d2-4f97-90ec-616c3d489f2f admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:05.711369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3e121-09d2-4f97-90ec-616c3d489f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:05.711642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3e121-09d2-4f97-90ec-616c3d489f2f admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:05.712267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3e121-09d2-4f97-90ec-616c3d489f2f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:05.714234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3e121-09d2-4f97-90ec-616c3d489f2f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:05.878695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1266/2551] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:05 2022] POST /identity/v3/users => generated 285 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:05.882724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8a88c1-9938-4d32-afc0-9019f526c4a3 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:05.895446 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8a88c1-9938-4d32-afc0-9019f526c4a3 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:05.900591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19bc27d-6521-4c07-b18c-3478349f7287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:05.901242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19bc27d-6521-4c07-b18c-3478349f7287 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:05.901520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19bc27d-6521-4c07-b18c-3478349f7287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:05.901822 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19bc27d-6521-4c07-b18c-3478349f7287 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:05.902470 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19bc27d-6521-4c07-b18c-3478349f7287 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:05.904183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19bc27d-6521-4c07-b18c-3478349f7287 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:05.916385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1286/2552] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:05.921211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a3e121-09d2-4f97-90ec-616c3d489f2f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:05.932301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a3e121-09d2-4f97-90ec-616c3d489f2f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:05.937257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e562d55-db9a-4cc3-97b6-5b26599c01b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:05.938198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e562d55-db9a-4cc3-97b6-5b26599c01b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:05.943507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e562d55-db9a-4cc3-97b6-5b26599c01b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:05.943507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e562d55-db9a-4cc3-97b6-5b26599c01b4 admin admin] PATH_INFO: `/v3/projects/8ee1719a15cc42329d5f996f4c14aaa4/users/6a4561dc99ce4323b10439693d2c54b6/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:05.955557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e562d55-db9a-4cc3-97b6-5b26599c01b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ee1719a15cc42329d5f996f4c14aaa4, user_id=6a4561dc99ce4323b10439693d2c54b6, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:05.958964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e562d55-db9a-4cc3-97b6-5b26599c01b4 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:06.031066 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1267/2553] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:05 2022] PUT /identity/v3/projects/8ee1719a15cc42329d5f996f4c14aaa4/users/6a4561dc99ce4323b10439693d2c54b6/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:06.035184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19bc27d-6521-4c07-b18c-3478349f7287 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:06.045247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19bc27d-6521-4c07-b18c-3478349f7287 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:06.108329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb18d00-4800-4f20-b43c-46260d8e9c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:06.109309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb18d00-4800-4f20-b43c-46260d8e9c1c admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:06.109309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb18d00-4800-4f20-b43c-46260d8e9c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:06.109556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb18d00-4800-4f20-b43c-46260d8e9c1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:06.110232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb18d00-4800-4f20-b43c-46260d8e9c1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:06.113030 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb18d00-4800-4f20-b43c-46260d8e9c1c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:06.123665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1287/2554] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:06 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:06.128419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e562d55-db9a-4cc3-97b6-5b26599c01b4 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:06.140166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e562d55-db9a-4cc3-97b6-5b26599c01b4 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:06.164038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:06.164038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:06.164038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:06.164038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] PATH_INFO: `/v3/projects/8ee1719a15cc42329d5f996f4c14aaa4/users/6a4561dc99ce4323b10439693d2c54b6/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:06.164782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ee1719a15cc42329d5f996f4c14aaa4, user_id=6a4561dc99ce4323b10439693d2c54b6, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:06.164782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:06.164782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1268/2555] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:06 2022] PUT /identity/v3/projects/8ee1719a15cc42329d5f996f4c14aaa4/users/6a4561dc99ce4323b10439693d2c54b6/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:06.175647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c9f4ac-10df-4d12-8d8a-583a4a6284e8 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:06.175939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c9f4ac-10df-4d12-8d8a-583a4a6284e8 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:06.176216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c9f4ac-10df-4d12-8d8a-583a4a6284e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:06.207922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-47c9f4ac-10df-4d12-8d8a-583a4a6284e8 None None] MFA Rules not processed for user `6a4561dc99ce4323b10439693d2c54b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:06.209151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-47c9f4ac-10df-4d12-8d8a-583a4a6284e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:06.214302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:06.289980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1288/2556] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:06 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:06.301851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:06.303595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:06.334387 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:06.361205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677a9c8c-73fc-4e94-88b2-849dc96a00d3 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:06.364902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:06.365502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:06.365812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:06.366083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:06.374870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:06.376301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:06.385553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1269/2557] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:06 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:06.650262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c9f4ac-10df-4d12-8d8a-583a4a6284e8 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:06.659780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c9f4ac-10df-4d12-8d8a-583a4a6284e8 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:06.663351 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:06.663976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:06.664258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:06.664535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:06.722339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:06.723955 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:06.732760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1289/2558] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:06 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:08.971030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:08.973344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:09.033198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6256ff4c-6c69-4b92-82d5-2855c49e78a8 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:09.037661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:09.038268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:09.038528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:09.038796 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:09.040557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:09.107088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:09.108515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:09.118224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1270/2559] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:12.679736 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:12.681253 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:12.696594 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ce485-7778-4e66-bb38-2a9db922cfdf service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:12.701568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:12.702270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:12.702551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:12.702830 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] PATH_INFO: `/v3/users/3326fcf3ccbc4a978e9e52e25456ad98` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:12.717438 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3326fcf3ccbc4a978e9e52e25456ad98)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:12.719416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:12.914271 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3326fcf3ccbc4a978e9e52e25456ad98'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:12.914942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3326fcf3ccbc4a978e9e52e25456ad98'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:12.927452 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3326fcf3ccbc4a978e9e52e25456ad98'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:13.003647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3326fcf3ccbc4a978e9e52e25456ad98'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:13.011228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f5220065-6554-495c-b196-a6e793b70432 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3326fcf3ccbc4a978e9e52e25456ad98'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:13.017772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1290/2560] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:12 2022] DELETE /identity/v3/users/3326fcf3ccbc4a978e9e52e25456ad98 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:13.261348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:13.272223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ed0c1f-e1ee-48e5-a4b2-a46e776a2322 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:13.364794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:13.365518 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:13.365888 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:13.366235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] PATH_INFO: `/v3/projects/dcd7d2bdaffb4d168f27cf6444e005dd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:13.380026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] RBAC: Authorizing `identity:delete_project(project_id=dcd7d2bdaffb4d168f27cf6444e005dd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:13.382267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:13.404964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:13.513642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dcd7d2bdaffb4d168f27cf6444e005dd', 'request_id': 'req-045e060a-6676-4879-a5d5-742e0c3a251d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:13.591404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dcd7d2bdaffb4d168f27cf6444e005dd', 'request_id': 'req-045e060a-6676-4879-a5d5-742e0c3a251d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:13.598989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dcd7d2bdaffb4d168f27cf6444e005dd', 'request_id': 'req-045e060a-6676-4879-a5d5-742e0c3a251d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:13.613671 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] The token cache is being invalidate because project dcd7d2bdaffb4d168f27cf6444e005dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:13.613958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:13.614610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-045e060a-6676-4879-a5d5-742e0c3a251d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:13.616192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1271/2561] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:13 2022] DELETE /identity/v3/projects/dcd7d2bdaffb4d168f27cf6444e005dd => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:13.621138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84900787-df41-4f59-a177-7d2dace6c196 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:13.621430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84900787-df41-4f59-a177-7d2dace6c196 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:13.621754 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84900787-df41-4f59-a177-7d2dace6c196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:13.629315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:13.644037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-84900787-df41-4f59-a177-7d2dace6c196 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:13.645260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-84900787-df41-4f59-a177-7d2dace6c196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:13.712024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1291/2562] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:13.719749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1e74a5-745c-4de8-a0ed-44779d9cc399 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:13.720131 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1e74a5-745c-4de8-a0ed-44779d9cc399 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:13.720511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1e74a5-745c-4de8-a0ed-44779d9cc399 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:13.742629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8f1e74a5-745c-4de8-a0ed-44779d9cc399 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:13.743907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8f1e74a5-745c-4de8-a0ed-44779d9cc399 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:13.757966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1272/2563] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:13.762300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84900787-df41-4f59-a177-7d2dace6c196 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:13.775553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84900787-df41-4f59-a177-7d2dace6c196 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:13.776979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:13.777573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:13.777882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:13.778245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:13.778918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:24:13.779221 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:13.780801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:13.792724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1292/2564] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:24:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:24:13.796991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1e74a5-745c-4de8-a0ed-44779d9cc399 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:13.806427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1e74a5-745c-4de8-a0ed-44779d9cc399 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:13.811175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3f4688-7e21-42d1-ac20-17a68bb58a6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:13.813454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3f4688-7e21-42d1-ac20-17a68bb58a6e admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:13.813454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3f4688-7e21-42d1-ac20-17a68bb58a6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:13.813454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3f4688-7e21-42d1-ac20-17a68bb58a6e admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:13.813454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3f4688-7e21-42d1-ac20-17a68bb58a6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:13.815133 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3f4688-7e21-42d1-ac20-17a68bb58a6e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:13.935427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1273/2565] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:13 2022] POST /identity/v3/projects => generated 405 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:13.939685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:13.950360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212941f6-ddf5-4bdd-b084-37826b24daf4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:14.008049 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1169d3ee-30c6-43e9-bde2-aaef8dfd4ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:14.008654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169d3ee-30c6-43e9-bde2-aaef8dfd4ac8 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:14.009028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169d3ee-30c6-43e9-bde2-aaef8dfd4ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:14.009307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169d3ee-30c6-43e9-bde2-aaef8dfd4ac8 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:14.010019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1169d3ee-30c6-43e9-bde2-aaef8dfd4ac8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:14.012241 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1169d3ee-30c6-43e9-bde2-aaef8dfd4ac8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:14.110440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1293/2566] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:13 2022] POST /identity/v3/users => generated 303 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:24:14.114702 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3f4688-7e21-42d1-ac20-17a68bb58a6e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:14.125313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3f4688-7e21-42d1-ac20-17a68bb58a6e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:14.130482 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfbda2f-9b9c-4688-97ba-667149edd076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:14.131079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfbda2f-9b9c-4688-97ba-667149edd076 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:14.131359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfbda2f-9b9c-4688-97ba-667149edd076 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:14.131631 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfbda2f-9b9c-4688-97ba-667149edd076 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:14.132374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfbda2f-9b9c-4688-97ba-667149edd076 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:14.134069 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfbda2f-9b9c-4688-97ba-667149edd076 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:14.145420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1274/2567] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:14.149608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1169d3ee-30c6-43e9-bde2-aaef8dfd4ac8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:14.159741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1169d3ee-30c6-43e9-bde2-aaef8dfd4ac8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:14.164423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfcf4d3-0495-4913-a8de-3034b93cc752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:14.165036 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfcf4d3-0495-4913-a8de-3034b93cc752 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:14.165320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfcf4d3-0495-4913-a8de-3034b93cc752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:14.165614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfcf4d3-0495-4913-a8de-3034b93cc752 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:14.166264 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfcf4d3-0495-4913-a8de-3034b93cc752 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:14.167864 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfcf4d3-0495-4913-a8de-3034b93cc752 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:14.178573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1294/2568] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:14.182893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfbda2f-9b9c-4688-97ba-667149edd076 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:14.193547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfbda2f-9b9c-4688-97ba-667149edd076 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:14.198173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:14.198818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:14.199094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:14.199364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] PATH_INFO: `/v3/projects/6e01d9e866464d5e917acc9e102a9e48/users/d60cb507244e48d08ea3fa0aa4f6fe80/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:14.215296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e01d9e866464d5e917acc9e102a9e48, user_id=d60cb507244e48d08ea3fa0aa4f6fe80, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:14.218175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:14.282770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1275/2569] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:14 2022] PUT /identity/v3/projects/6e01d9e866464d5e917acc9e102a9e48/users/d60cb507244e48d08ea3fa0aa4f6fe80/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:14.288015 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b971a503-8c90-4cde-92f1-2d3806718b90 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:14.288323 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b971a503-8c90-4cde-92f1-2d3806718b90 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:14.288623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b971a503-8c90-4cde-92f1-2d3806718b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:14.308450 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b971a503-8c90-4cde-92f1-2d3806718b90 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:14.313019 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b971a503-8c90-4cde-92f1-2d3806718b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:14.371010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1295/2570] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:24:14.385902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:14.388668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:14.447446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14435d75-caa4-4e0f-b40c-39e250345476 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:14.451058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f3d70-c058-4625-902d-a1e5d6f06a40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:14.451664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f3d70-c058-4625-902d-a1e5d6f06a40 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:14.451946 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f3d70-c058-4625-902d-a1e5d6f06a40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:14.452244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08f3d70-c058-4625-902d-a1e5d6f06a40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:14.461156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08f3d70-c058-4625-902d-a1e5d6f06a40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:14.462776 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08f3d70-c058-4625-902d-a1e5d6f06a40 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:14.472425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1276/2571] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:15.831896 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24296081-e551-485d-bb37-0dbb13dd262c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:15.832459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24296081-e551-485d-bb37-0dbb13dd262c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:15.832459 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24296081-e551-485d-bb37-0dbb13dd262c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:15.852744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:15.862224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-24296081-e551-485d-bb37-0dbb13dd262c None None] MFA Rules not processed for user `d60cb507244e48d08ea3fa0aa4f6fe80`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:15.863514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-24296081-e551-485d-bb37-0dbb13dd262c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:15.932481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1296/2572] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:15 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:15.937084 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f3d70-c058-4625-902d-a1e5d6f06a40 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:15.947489 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08f3d70-c058-4625-902d-a1e5d6f06a40 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:15.952173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63e9730-4b79-493d-b8de-5ca436b04f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:15.952768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e9730-4b79-493d-b8de-5ca436b04f38 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:15.953049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e9730-4b79-493d-b8de-5ca436b04f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:15.953317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63e9730-4b79-493d-b8de-5ca436b04f38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:15.953973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63e9730-4b79-493d-b8de-5ca436b04f38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:15.955897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63e9730-4b79-493d-b8de-5ca436b04f38 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:16.070793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1277/2573] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:15 2022] POST /identity/v3/projects => generated 403 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:16.072274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24296081-e551-485d-bb37-0dbb13dd262c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:16.084634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24296081-e551-485d-bb37-0dbb13dd262c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:16.146224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3b2fd-b702-4148-95d9-181a107c3058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:16.146838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3b2fd-b702-4148-95d9-181a107c3058 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:16.147113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3b2fd-b702-4148-95d9-181a107c3058 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:16.147380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d3b2fd-b702-4148-95d9-181a107c3058 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:16.148014 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d3b2fd-b702-4148-95d9-181a107c3058 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:16.150042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d3b2fd-b702-4148-95d9-181a107c3058 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:16.246260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1297/2574] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:16 2022] POST /identity/v3/users => generated 308 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:16.250445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63e9730-4b79-493d-b8de-5ca436b04f38 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:16.260706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63e9730-4b79-493d-b8de-5ca436b04f38 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:16.265247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339a2f07-41b8-4a78-ad9f-1af7f4a623b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:16.266049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339a2f07-41b8-4a78-ad9f-1af7f4a623b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:16.266333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339a2f07-41b8-4a78-ad9f-1af7f4a623b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:16.266704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339a2f07-41b8-4a78-ad9f-1af7f4a623b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:16.267487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339a2f07-41b8-4a78-ad9f-1af7f4a623b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:16.269430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339a2f07-41b8-4a78-ad9f-1af7f4a623b7 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:16.280648 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1278/2575] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:16.285484 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3b2fd-b702-4148-95d9-181a107c3058 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:16.295942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d3b2fd-b702-4148-95d9-181a107c3058 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:16.301235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f869465-4b8b-421c-80ae-3dd7dbb25d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:16.301915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f869465-4b8b-421c-80ae-3dd7dbb25d43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:16.302189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f869465-4b8b-421c-80ae-3dd7dbb25d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:16.302457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f869465-4b8b-421c-80ae-3dd7dbb25d43 admin admin] PATH_INFO: `/v3/projects/096647a7ac304316a3ef24675981d416/users/af959d52de8644d3bd219f01d6cdb34e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:16.318054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f869465-4b8b-421c-80ae-3dd7dbb25d43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=096647a7ac304316a3ef24675981d416, user_id=af959d52de8644d3bd219f01d6cdb34e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:16.320694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f869465-4b8b-421c-80ae-3dd7dbb25d43 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:16.402714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1298/2576] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:16 2022] PUT /identity/v3/projects/096647a7ac304316a3ef24675981d416/users/af959d52de8644d3bd219f01d6cdb34e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:16.406637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339a2f07-41b8-4a78-ad9f-1af7f4a623b7 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:16.417139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339a2f07-41b8-4a78-ad9f-1af7f4a623b7 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:16.421069 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:16.477088 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ddc8dc-fef6-48f1-82a7-d23d73dfd913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:16.477737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddc8dc-fef6-48f1-82a7-d23d73dfd913 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:16.478018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddc8dc-fef6-48f1-82a7-d23d73dfd913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:16.478322 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddc8dc-fef6-48f1-82a7-d23d73dfd913 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:16.478923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ddc8dc-fef6-48f1-82a7-d23d73dfd913 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:16.480774 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ddc8dc-fef6-48f1-82a7-d23d73dfd913 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:16.495613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1279/2577] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:16 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:16.499561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f869465-4b8b-421c-80ae-3dd7dbb25d43 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:16.510133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f869465-4b8b-421c-80ae-3dd7dbb25d43 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:16.514985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89ab36d-e92c-461c-beb3-23a510f055c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:16.515502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89ab36d-e92c-461c-beb3-23a510f055c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:16.515777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89ab36d-e92c-461c-beb3-23a510f055c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:16.516050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89ab36d-e92c-461c-beb3-23a510f055c5 admin admin] PATH_INFO: `/v3/projects/096647a7ac304316a3ef24675981d416/users/af959d52de8644d3bd219f01d6cdb34e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:16.518462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89ab36d-e92c-461c-beb3-23a510f055c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=096647a7ac304316a3ef24675981d416, user_id=af959d52de8644d3bd219f01d6cdb34e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:16.521158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89ab36d-e92c-461c-beb3-23a510f055c5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:16.536042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1299/2578] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:16 2022] PUT /identity/v3/projects/096647a7ac304316a3ef24675981d416/users/af959d52de8644d3bd219f01d6cdb34e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:18.193513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984c79a-6889-411d-ae0e-e29f349bba09 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:18.194018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984c79a-6889-411d-ae0e-e29f349bba09 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:18.194284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984c79a-6889-411d-ae0e-e29f349bba09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:18.225970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-6984c79a-6889-411d-ae0e-e29f349bba09 None None] MFA Rules not processed for user `af959d52de8644d3bd219f01d6cdb34e`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:18.230018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6984c79a-6889-411d-ae0e-e29f349bba09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:18.308437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1280/2579] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:18 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:19.717208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89ab36d-e92c-461c-beb3-23a510f055c5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:19.769477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:19.780492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89ab36d-e92c-461c-beb3-23a510f055c5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:19.785037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:19.785686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:19.785966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:19.786235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] PATH_INFO: `/v3/users/d60cb507244e48d08ea3fa0aa4f6fe80` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:19.787339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d60cb507244e48d08ea3fa0aa4f6fe80)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:19.789233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:20.032663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd60cb507244e48d08ea3fa0aa4f6fe80'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.033819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd60cb507244e48d08ea3fa0aa4f6fe80'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.045347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd60cb507244e48d08ea3fa0aa4f6fe80'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.132004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd60cb507244e48d08ea3fa0aa4f6fe80'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.139670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd60cb507244e48d08ea3fa0aa4f6fe80'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.146554 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1300/2580] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:19 2022] DELETE /identity/v3/users/d60cb507244e48d08ea3fa0aa4f6fe80 => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:20.432549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984c79a-6889-411d-ae0e-e29f349bba09 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:20.447715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984c79a-6889-411d-ae0e-e29f349bba09 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:20.503860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:20.504491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:20.504821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:20.505098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] PATH_INFO: `/v3/projects/6e01d9e866464d5e917acc9e102a9e48` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:20.506231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e01d9e866464d5e917acc9e102a9e48)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:20.508134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:20.659515 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:20.671549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e01d9e866464d5e917acc9e102a9e48', 'request_id': 'req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.745490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e01d9e866464d5e917acc9e102a9e48', 'request_id': 'req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.751841 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e01d9e866464d5e917acc9e102a9e48', 'request_id': 'req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.763158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] The token cache is being invalidate because project 6e01d9e866464d5e917acc9e102a9e48 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:20.763446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.764109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:20.765672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1281/2581] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:20 2022] DELETE /identity/v3/projects/6e01d9e866464d5e917acc9e102a9e48 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:20.769630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:20.770999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:20.784089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da39dac-be44-455d-8aa9-eddc6b997157 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:20.838947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:20.839586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:20.839862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:20.840153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] PATH_INFO: `/v3/users/af959d52de8644d3bd219f01d6cdb34e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:20.841322 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af959d52de8644d3bd219f01d6cdb34e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:20.843303 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:21.067476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af959d52de8644d3bd219f01d6cdb34e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.068120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af959d52de8644d3bd219f01d6cdb34e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.080328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af959d52de8644d3bd219f01d6cdb34e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.145076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af959d52de8644d3bd219f01d6cdb34e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.146731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:21.153671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4adc8dd5-0d82-4ec6-9985-91aad92172f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af959d52de8644d3bd219f01d6cdb34e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.160114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1301/2582] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:20 2022] DELETE /identity/v3/users/af959d52de8644d3bd219f01d6cdb34e => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:21.409713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:21.411162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:21.423894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7bea6f-be05-40bd-9ec9-56c1b808c0c5 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:21.488678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:21.489394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:21.489666 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:21.489942 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] PATH_INFO: `/v3/projects/096647a7ac304316a3ef24675981d416` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:21.491071 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] RBAC: Authorizing `identity:delete_project(project_id=096647a7ac304316a3ef24675981d416)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:21.493013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:21.631651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '096647a7ac304316a3ef24675981d416', 'request_id': 'req-d9e7e1ad-3f25-492a-9e04-f4933cd823be'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.733394 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '096647a7ac304316a3ef24675981d416', 'request_id': 'req-d9e7e1ad-3f25-492a-9e04-f4933cd823be'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.742202 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '096647a7ac304316a3ef24675981d416', 'request_id': 'req-d9e7e1ad-3f25-492a-9e04-f4933cd823be'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.752200 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] The token cache is being invalidate because project 096647a7ac304316a3ef24675981d416 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:21.753031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.753713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d9e7e1ad-3f25-492a-9e04-f4933cd823be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:21.755347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1282/2583] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:21 2022] DELETE /identity/v3/projects/096647a7ac304316a3ef24675981d416 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:21.760105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41153da4-4285-493c-acea-6c358e70a897 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:21.760394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41153da4-4285-493c-acea-6c358e70a897 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:21.760684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41153da4-4285-493c-acea-6c358e70a897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:21.782685 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-41153da4-4285-493c-acea-6c358e70a897 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:21.783935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-41153da4-4285-493c-acea-6c358e70a897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:21.858558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1302/2584] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:21.866784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cc6532-3f0f-4e8e-a252-b17e56cb8195 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:21.867091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cc6532-3f0f-4e8e-a252-b17e56cb8195 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:21.867368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cc6532-3f0f-4e8e-a252-b17e56cb8195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:21.887183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-56cc6532-3f0f-4e8e-a252-b17e56cb8195 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:21.888379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-56cc6532-3f0f-4e8e-a252-b17e56cb8195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:21.896310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1283/2585] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:21.900569 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41153da4-4285-493c-acea-6c358e70a897 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:21.910313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41153da4-4285-493c-acea-6c358e70a897 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:21.915011 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:21.915586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:21.915845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:21.916111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:21.916744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:24:21.917040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:21.918652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:21.930587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1303/2586] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:24:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:24:21.935258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cc6532-3f0f-4e8e-a252-b17e56cb8195 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:21.945359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cc6532-3f0f-4e8e-a252-b17e56cb8195 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:21.950158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ee0c3b-e56f-4601-af1b-94295be69294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:21.950868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ee0c3b-e56f-4601-af1b-94295be69294 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:21.951073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ee0c3b-e56f-4601-af1b-94295be69294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:21.951338 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ee0c3b-e56f-4601-af1b-94295be69294 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:21.951974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ee0c3b-e56f-4601-af1b-94295be69294 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:21.953911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ee0c3b-e56f-4601-af1b-94295be69294 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:22.055425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1284/2587] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:21 2022] POST /identity/v3/projects => generated 385 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:22.059691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:22.070591 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa6bcd4-42f7-445f-b48f-aa937b8c66ee admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:22.126032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cabf35-9c86-44ee-85c9-d7643efe83f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:22.126643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cabf35-9c86-44ee-85c9-d7643efe83f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.126920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cabf35-9c86-44ee-85c9-d7643efe83f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.127192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cabf35-9c86-44ee-85c9-d7643efe83f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.128425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cabf35-9c86-44ee-85c9-d7643efe83f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:22.130372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cabf35-9c86-44ee-85c9-d7643efe83f9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:22.218518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1304/2588] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:22 2022] POST /identity/v3/users => generated 293 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:22.223521 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ee0c3b-e56f-4601-af1b-94295be69294 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:22.234233 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ee0c3b-e56f-4601-af1b-94295be69294 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:22.239502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5688c4-f634-4a3f-93ef-e50282443fbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:22.240102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5688c4-f634-4a3f-93ef-e50282443fbf admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.240378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5688c4-f634-4a3f-93ef-e50282443fbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.240649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5688c4-f634-4a3f-93ef-e50282443fbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.241309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5688c4-f634-4a3f-93ef-e50282443fbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:22.243006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5688c4-f634-4a3f-93ef-e50282443fbf admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:22.252918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1285/2589] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:22.256912 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cabf35-9c86-44ee-85c9-d7643efe83f9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:22.267067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cabf35-9c86-44ee-85c9-d7643efe83f9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:22.271747 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1156077-5fab-41e6-8efd-1700840bec7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:22.272342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1156077-5fab-41e6-8efd-1700840bec7a admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.272624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1156077-5fab-41e6-8efd-1700840bec7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.272900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1156077-5fab-41e6-8efd-1700840bec7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.273541 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1156077-5fab-41e6-8efd-1700840bec7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:22.275163 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1156077-5fab-41e6-8efd-1700840bec7a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:22.286123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1305/2590] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:22.290074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5688c4-f634-4a3f-93ef-e50282443fbf admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:22.300429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5688c4-f634-4a3f-93ef-e50282443fbf admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:22.305074 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8af562e-cad7-4cfa-8bc0-0ac8184887cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:22.305753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8af562e-cad7-4cfa-8bc0-0ac8184887cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.306039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8af562e-cad7-4cfa-8bc0-0ac8184887cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.306313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8af562e-cad7-4cfa-8bc0-0ac8184887cc admin admin] PATH_INFO: `/v3/projects/e252c14108a0446993dd6c3a41da889b/users/67270b96f21d49058390bc661d981234/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.309264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:22.320480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8af562e-cad7-4cfa-8bc0-0ac8184887cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=e252c14108a0446993dd6c3a41da889b, user_id=67270b96f21d49058390bc661d981234, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:22.323150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8af562e-cad7-4cfa-8bc0-0ac8184887cc admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:22.425466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1286/2591] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:22 2022] PUT /identity/v3/projects/e252c14108a0446993dd6c3a41da889b/users/67270b96f21d49058390bc661d981234/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:22.437230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e954eb40-ff09-4442-9a8e-6ac813b20d25 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.437514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e954eb40-ff09-4442-9a8e-6ac813b20d25 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.437819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e954eb40-ff09-4442-9a8e-6ac813b20d25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.465734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e954eb40-ff09-4442-9a8e-6ac813b20d25 None None] MFA Rules not processed for user `67270b96f21d49058390bc661d981234`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:22.466929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e954eb40-ff09-4442-9a8e-6ac813b20d25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:22.510489 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:22.531147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1306/2592] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:22.541751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8af562e-cad7-4cfa-8bc0-0ac8184887cc admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:22.601847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8af562e-cad7-4cfa-8bc0-0ac8184887cc admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:22.606452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15168d4b-e6ed-4acc-ac76-fa6c023b7897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:22.607047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15168d4b-e6ed-4acc-ac76-fa6c023b7897 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.607323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15168d4b-e6ed-4acc-ac76-fa6c023b7897 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.607592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15168d4b-e6ed-4acc-ac76-fa6c023b7897 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.608210 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15168d4b-e6ed-4acc-ac76-fa6c023b7897 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:22.610111 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15168d4b-e6ed-4acc-ac76-fa6c023b7897 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:22.717097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1287/2593] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:22 2022] POST /identity/v3/projects => generated 385 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:22.721050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e954eb40-ff09-4442-9a8e-6ac813b20d25 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:22.732645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e954eb40-ff09-4442-9a8e-6ac813b20d25 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:22.789793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19c18c2-4542-402f-8e9d-1350f07c3d65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:22.790445 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19c18c2-4542-402f-8e9d-1350f07c3d65 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.790668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19c18c2-4542-402f-8e9d-1350f07c3d65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.790942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19c18c2-4542-402f-8e9d-1350f07c3d65 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.791566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19c18c2-4542-402f-8e9d-1350f07c3d65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:22.793483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19c18c2-4542-402f-8e9d-1350f07c3d65 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:22.904422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1307/2594] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:22 2022] POST /identity/v3/users => generated 299 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:22.908381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15168d4b-e6ed-4acc-ac76-fa6c023b7897 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:22.918905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15168d4b-e6ed-4acc-ac76-fa6c023b7897 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:22.923526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81342926-0df6-48e7-88bb-5e9c8fa629e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:22.924122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81342926-0df6-48e7-88bb-5e9c8fa629e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.924402 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81342926-0df6-48e7-88bb-5e9c8fa629e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.924723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81342926-0df6-48e7-88bb-5e9c8fa629e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.925342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81342926-0df6-48e7-88bb-5e9c8fa629e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:22.927031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81342926-0df6-48e7-88bb-5e9c8fa629e9 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:22.937329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1288/2595] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:22.941804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19c18c2-4542-402f-8e9d-1350f07c3d65 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:22.951755 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19c18c2-4542-402f-8e9d-1350f07c3d65 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:22.956989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf7ef0b-cf3f-404f-aca5-1ad9dc43d300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:22.957650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf7ef0b-cf3f-404f-aca5-1ad9dc43d300 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:22.957930 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf7ef0b-cf3f-404f-aca5-1ad9dc43d300 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:22.958200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf7ef0b-cf3f-404f-aca5-1ad9dc43d300 admin admin] PATH_INFO: `/v3/projects/bed75bfb31d74f6e8dfbe54d3c3d688d/users/2c95bcbb965349c189098d3e62a3c36b/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:22.977523 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf7ef0b-cf3f-404f-aca5-1ad9dc43d300 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bed75bfb31d74f6e8dfbe54d3c3d688d, user_id=2c95bcbb965349c189098d3e62a3c36b, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:22.980124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf7ef0b-cf3f-404f-aca5-1ad9dc43d300 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:23.065088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1308/2596] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:22 2022] PUT /identity/v3/projects/bed75bfb31d74f6e8dfbe54d3c3d688d/users/2c95bcbb965349c189098d3e62a3c36b/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:24:23.069500 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81342926-0df6-48e7-88bb-5e9c8fa629e9 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:23.079532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81342926-0df6-48e7-88bb-5e9c8fa629e9 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:23.136152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c154077c-87f7-4c13-a569-f4f6daeac936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:23.136754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154077c-87f7-4c13-a569-f4f6daeac936 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:23.137034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154077c-87f7-4c13-a569-f4f6daeac936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:23.137309 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154077c-87f7-4c13-a569-f4f6daeac936 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:23.138015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c154077c-87f7-4c13-a569-f4f6daeac936 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:23.139599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c154077c-87f7-4c13-a569-f4f6daeac936 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:23.149812 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1289/2597] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:23 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:23.153947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf7ef0b-cf3f-404f-aca5-1ad9dc43d300 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:23.164203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf7ef0b-cf3f-404f-aca5-1ad9dc43d300 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:23.168795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:23.169430 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:23.169812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:23.170084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] PATH_INFO: `/v3/projects/bed75bfb31d74f6e8dfbe54d3c3d688d/users/2c95bcbb965349c189098d3e62a3c36b/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:23.171978 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bed75bfb31d74f6e8dfbe54d3c3d688d, user_id=2c95bcbb965349c189098d3e62a3c36b, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:23.174739 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:23.184539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1309/2598] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:23 2022] PUT /identity/v3/projects/bed75bfb31d74f6e8dfbe54d3c3d688d/users/2c95bcbb965349c189098d3e62a3c36b/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:23.195038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cd1030-9718-4816-96d4-d812ca11bf3f None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:23.195344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cd1030-9718-4816-96d4-d812ca11bf3f None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:23.195624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95cd1030-9718-4816-96d4-d812ca11bf3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:23.223616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-95cd1030-9718-4816-96d4-d812ca11bf3f None None] MFA Rules not processed for user `2c95bcbb965349c189098d3e62a3c36b`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:23.224825 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-95cd1030-9718-4816-96d4-d812ca11bf3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:23.269134 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:23.292751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1290/2599] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:23 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:23.304329 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:23.306050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:23.363847 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b931872-08f1-4d17-a8b6-e79556502340 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:23.367506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa4bf3c-5560-4e5d-b8da-c4f3ad4d6fb7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:23.368105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa4bf3c-5560-4e5d-b8da-c4f3ad4d6fb7 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:23.368381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa4bf3c-5560-4e5d-b8da-c4f3ad4d6fb7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:23.368654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa4bf3c-5560-4e5d-b8da-c4f3ad4d6fb7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:23.377514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa4bf3c-5560-4e5d-b8da-c4f3ad4d6fb7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:23.378987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa4bf3c-5560-4e5d-b8da-c4f3ad4d6fb7 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:23.388330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1310/2600] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:23 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:23.416577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cd1030-9718-4816-96d4-d812ca11bf3f None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:23.477135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95cd1030-9718-4816-96d4-d812ca11bf3f None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:23.481723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:23.482347 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:23.482630 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:23.482901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] PATH_INFO: `/v3/users/67270b96f21d49058390bc661d981234` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:23.484028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=67270b96f21d49058390bc661d981234)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:23.485968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:23.729850 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '67270b96f21d49058390bc661d981234'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:23.741649 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '67270b96f21d49058390bc661d981234'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:23.742389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '67270b96f21d49058390bc661d981234'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:23.818937 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '67270b96f21d49058390bc661d981234'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:23.825943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '67270b96f21d49058390bc661d981234'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:23.832409 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1291/2601] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:23 2022] DELETE /identity/v3/users/67270b96f21d49058390bc661d981234 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:23.836860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4e688e-70f8-4038-b925-ccdd77a8273b None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:23.837306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4e688e-70f8-4038-b925-ccdd77a8273b None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:23.837427 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4e688e-70f8-4038-b925-ccdd77a8273b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:23.855918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4f4e688e-70f8-4038-b925-ccdd77a8273b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:23.857134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4f4e688e-70f8-4038-b925-ccdd77a8273b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:23.891144 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:23.918979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1311/2602] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:23.928843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:23.932035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:23.996566 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8d5778-0da8-486f-a695-283a0fba5bf3 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:24.000288 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5201009-08af-4a45-bb84-25a47dae9cac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:24.000892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5201009-08af-4a45-bb84-25a47dae9cac service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:24.001173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5201009-08af-4a45-bb84-25a47dae9cac service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:24.001512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5201009-08af-4a45-bb84-25a47dae9cac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:24.013756 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5201009-08af-4a45-bb84-25a47dae9cac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:24.015184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5201009-08af-4a45-bb84-25a47dae9cac service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:24.027097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1292/2603] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:24.438252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4e688e-70f8-4038-b925-ccdd77a8273b None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:24.439764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4f4e688e-70f8-4038-b925-ccdd77a8273b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:24.454986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4e688e-70f8-4038-b925-ccdd77a8273b None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:24.461336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:24.461969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:24.462255 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:24.462556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] PATH_INFO: `/v3/projects/e252c14108a0446993dd6c3a41da889b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:24.463644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e252c14108a0446993dd6c3a41da889b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:24.465635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:24.602402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e252c14108a0446993dd6c3a41da889b', 'request_id': 'req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:24.607994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e252c14108a0446993dd6c3a41da889b', 'request_id': 'req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:24.698816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e252c14108a0446993dd6c3a41da889b', 'request_id': 'req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:24.710765 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] The token cache is being invalidate because project e252c14108a0446993dd6c3a41da889b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:24.715697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:24.715697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:24.715697 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1312/2604] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:24 2022] DELETE /identity/v3/projects/e252c14108a0446993dd6c3a41da889b => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:24.717223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5201009-08af-4a45-bb84-25a47dae9cac service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:24.722097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:24.727682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5201009-08af-4a45-bb84-25a47dae9cac service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:24.783118 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:24.783787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:24.784061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:24.784330 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] PATH_INFO: `/v3/users/2c95bcbb965349c189098d3e62a3c36b` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:24.785425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c95bcbb965349c189098d3e62a3c36b)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:24.787358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:25.055279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c95bcbb965349c189098d3e62a3c36b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.067142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c95bcbb965349c189098d3e62a3c36b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.067808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c95bcbb965349c189098d3e62a3c36b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.180221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c95bcbb965349c189098d3e62a3c36b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.187723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-0b92a792-3eb2-440d-bafb-8bbb8283c75b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c95bcbb965349c189098d3e62a3c36b'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.194830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1293/2605] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:24 2022] DELETE /identity/v3/users/2c95bcbb965349c189098d3e62a3c36b => generated 0 bytes in 478 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:25.638072 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:25.639528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:25.653320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd4e548-93d5-43ae-8511-a8a33e4d3ff8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:25.713463 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:25.714054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:25.714333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:25.714606 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] PATH_INFO: `/v3/projects/bed75bfb31d74f6e8dfbe54d3c3d688d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:25.715720 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bed75bfb31d74f6e8dfbe54d3c3d688d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:25.717635 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:25.830600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bed75bfb31d74f6e8dfbe54d3c3d688d', 'request_id': 'req-fa0659d4-eff1-436b-8812-6754ba787331'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.836509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bed75bfb31d74f6e8dfbe54d3c3d688d', 'request_id': 'req-fa0659d4-eff1-436b-8812-6754ba787331'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.933135 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bed75bfb31d74f6e8dfbe54d3c3d688d', 'request_id': 'req-fa0659d4-eff1-436b-8812-6754ba787331'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.934483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:25.945352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] The token cache is being invalidate because project bed75bfb31d74f6e8dfbe54d3c3d688d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:25.945672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.946290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-fa0659d4-eff1-436b-8812-6754ba787331 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:25.947762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1313/2606] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:25 2022] DELETE /identity/v3/projects/bed75bfb31d74f6e8dfbe54d3c3d688d => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:25.952789 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081aa465-8998-4f13-9dfc-13afa1f00cc1 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:25.953270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081aa465-8998-4f13-9dfc-13afa1f00cc1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:25.953400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081aa465-8998-4f13-9dfc-13afa1f00cc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:25.973914 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-081aa465-8998-4f13-9dfc-13afa1f00cc1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:25.975395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-081aa465-8998-4f13-9dfc-13afa1f00cc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:26.039308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1294/2607] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:26.047204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3aff53-4292-45b0-8b09-19d7fbd3f59a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.047744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3aff53-4292-45b0-8b09-19d7fbd3f59a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.048192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3aff53-4292-45b0-8b09-19d7fbd3f59a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.068289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-2b3aff53-4292-45b0-8b09-19d7fbd3f59a None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:26.069473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2b3aff53-4292-45b0-8b09-19d7fbd3f59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:26.077283 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1314/2608] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:26.082158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081aa465-8998-4f13-9dfc-13afa1f00cc1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:26.091875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081aa465-8998-4f13-9dfc-13afa1f00cc1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:26.100560 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:26.101169 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.101449 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.101755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.102401 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:24:26.102699 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:26.104294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:26.115953 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1295/2609] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:24:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:24:26.120425 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3aff53-4292-45b0-8b09-19d7fbd3f59a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:26.130334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3aff53-4292-45b0-8b09-19d7fbd3f59a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:26.135034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe6e27e-292e-4e37-8ae1-e960521b68fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:26.135634 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6e27e-292e-4e37-8ae1-e960521b68fe admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.135911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6e27e-292e-4e37-8ae1-e960521b68fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.136182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe6e27e-292e-4e37-8ae1-e960521b68fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.136813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe6e27e-292e-4e37-8ae1-e960521b68fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:26.138742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe6e27e-292e-4e37-8ae1-e960521b68fe admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:26.261288 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1315/2610] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:26 2022] POST /identity/v3/projects => generated 391 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:26.265514 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:26.275591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de3f4c5-aca1-4fe7-91ec-9578fd4b8d00 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:26.280163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:26.331786 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bdb438-cf80-486c-9605-ac8d17bd9abd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:26.332416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bdb438-cf80-486c-9605-ac8d17bd9abd admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.332696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bdb438-cf80-486c-9605-ac8d17bd9abd admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.332970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bdb438-cf80-486c-9605-ac8d17bd9abd admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.333623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bdb438-cf80-486c-9605-ac8d17bd9abd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:26.335526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bdb438-cf80-486c-9605-ac8d17bd9abd admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:26.436963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1296/2611] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:26 2022] POST /identity/v3/users => generated 296 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:26.441272 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe6e27e-292e-4e37-8ae1-e960521b68fe admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:26.451794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe6e27e-292e-4e37-8ae1-e960521b68fe admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:26.456784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aa9554-500f-4da2-b079-556b6a155724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:26.457391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa9554-500f-4da2-b079-556b6a155724 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.458235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa9554-500f-4da2-b079-556b6a155724 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.458508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa9554-500f-4da2-b079-556b6a155724 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.459146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7aa9554-500f-4da2-b079-556b6a155724 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:26.460726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7aa9554-500f-4da2-b079-556b6a155724 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:26.470927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1316/2612] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:26.476178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bdb438-cf80-486c-9605-ac8d17bd9abd admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:26.485952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bdb438-cf80-486c-9605-ac8d17bd9abd admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:26.490897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d2b60c-4144-4cc7-ad5f-e8827dfb9401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:26.491486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d2b60c-4144-4cc7-ad5f-e8827dfb9401 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.491763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d2b60c-4144-4cc7-ad5f-e8827dfb9401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.492054 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d2b60c-4144-4cc7-ad5f-e8827dfb9401 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.492697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d2b60c-4144-4cc7-ad5f-e8827dfb9401 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:26.494306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d2b60c-4144-4cc7-ad5f-e8827dfb9401 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:26.504342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1297/2613] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:26 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:26.508379 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aa9554-500f-4da2-b079-556b6a155724 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:26.518318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aa9554-500f-4da2-b079-556b6a155724 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:26.523474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deee9b1b-b82a-4a99-abcf-9d9517faabc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:26.528125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deee9b1b-b82a-4a99-abcf-9d9517faabc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.528125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deee9b1b-b82a-4a99-abcf-9d9517faabc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.528125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deee9b1b-b82a-4a99-abcf-9d9517faabc9 admin admin] PATH_INFO: `/v3/projects/ef24ba5d0a144b688312fb238cf0ebe6/users/40e345e2deec435a953b99f01c7049c1/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.539854 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deee9b1b-b82a-4a99-abcf-9d9517faabc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef24ba5d0a144b688312fb238cf0ebe6, user_id=40e345e2deec435a953b99f01c7049c1, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:26.542526 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deee9b1b-b82a-4a99-abcf-9d9517faabc9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:26.619401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1317/2614] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:26 2022] PUT /identity/v3/projects/ef24ba5d0a144b688312fb238cf0ebe6/users/40e345e2deec435a953b99f01c7049c1/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:24:26.630907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecdb643-98ee-49c3-9fad-d42465f03593 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.631353 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecdb643-98ee-49c3-9fad-d42465f03593 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.631677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecdb643-98ee-49c3-9fad-d42465f03593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.660158 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8ecdb643-98ee-49c3-9fad-d42465f03593 None None] MFA Rules not processed for user `40e345e2deec435a953b99f01c7049c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:26.661598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8ecdb643-98ee-49c3-9fad-d42465f03593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:26.729301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1298/2615] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:26 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:26.733678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deee9b1b-b82a-4a99-abcf-9d9517faabc9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:26.794975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deee9b1b-b82a-4a99-abcf-9d9517faabc9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:26.799758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec6dd9d-d87d-49f7-8aa7-d71d245133b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:26.800422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6dd9d-d87d-49f7-8aa7-d71d245133b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:26.800706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6dd9d-d87d-49f7-8aa7-d71d245133b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:26.800983 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6dd9d-d87d-49f7-8aa7-d71d245133b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:26.801653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec6dd9d-d87d-49f7-8aa7-d71d245133b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:26.803573 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec6dd9d-d87d-49f7-8aa7-d71d245133b5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:26.935172 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1318/2616] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:26 2022] POST /identity/v3/projects => generated 393 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:26.939191 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecdb643-98ee-49c3-9fad-d42465f03593 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:26.950130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecdb643-98ee-49c3-9fad-d42465f03593 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:27.008224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf1d28c-7ddc-46bb-a173-aed812f3d243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:27.008877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf1d28c-7ddc-46bb-a173-aed812f3d243 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:27.009025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf1d28c-7ddc-46bb-a173-aed812f3d243 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:27.009297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf1d28c-7ddc-46bb-a173-aed812f3d243 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:27.009952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf1d28c-7ddc-46bb-a173-aed812f3d243 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:27.011862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf1d28c-7ddc-46bb-a173-aed812f3d243 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:27.112793 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1299/2617] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:26 2022] POST /identity/v3/users => generated 303 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:27.116586 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec6dd9d-d87d-49f7-8aa7-d71d245133b5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:27.127698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec6dd9d-d87d-49f7-8aa7-d71d245133b5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:27.133361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0307f5b7-f105-4f66-8e8a-1053b00cbe24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:27.134031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307f5b7-f105-4f66-8e8a-1053b00cbe24 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:27.134306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307f5b7-f105-4f66-8e8a-1053b00cbe24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:27.136032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0307f5b7-f105-4f66-8e8a-1053b00cbe24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:27.136032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0307f5b7-f105-4f66-8e8a-1053b00cbe24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:27.136833 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0307f5b7-f105-4f66-8e8a-1053b00cbe24 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:27.147861 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1319/2618] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:27.152244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf1d28c-7ddc-46bb-a173-aed812f3d243 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:27.162836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf1d28c-7ddc-46bb-a173-aed812f3d243 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:27.167656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d953ac9c-76c1-485e-98ef-5ecbe37f8b0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:27.168313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d953ac9c-76c1-485e-98ef-5ecbe37f8b0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:27.168595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d953ac9c-76c1-485e-98ef-5ecbe37f8b0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:27.168870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d953ac9c-76c1-485e-98ef-5ecbe37f8b0a admin admin] PATH_INFO: `/v3/projects/0461cf8c6e9242289e67ed2fa8819749/users/0f1b24ef8c644d35a74de8c51d9aba3b/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:27.171189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:27.184547 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d953ac9c-76c1-485e-98ef-5ecbe37f8b0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0461cf8c6e9242289e67ed2fa8819749, user_id=0f1b24ef8c644d35a74de8c51d9aba3b, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:27.187219 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d953ac9c-76c1-485e-98ef-5ecbe37f8b0a admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:27.271183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1300/2619] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:27 2022] PUT /identity/v3/projects/0461cf8c6e9242289e67ed2fa8819749/users/0f1b24ef8c644d35a74de8c51d9aba3b/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:27.275068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0307f5b7-f105-4f66-8e8a-1053b00cbe24 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:27.285012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0307f5b7-f105-4f66-8e8a-1053b00cbe24 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:27.340105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2e4fc3-e35f-46c1-8de2-8b6998359fde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:27.340706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2e4fc3-e35f-46c1-8de2-8b6998359fde admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:27.340987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2e4fc3-e35f-46c1-8de2-8b6998359fde admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:27.341261 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2e4fc3-e35f-46c1-8de2-8b6998359fde admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:27.341927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2e4fc3-e35f-46c1-8de2-8b6998359fde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:27.343543 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2e4fc3-e35f-46c1-8de2-8b6998359fde admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:27.349318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:27.354542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1320/2620] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:27.358642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d953ac9c-76c1-485e-98ef-5ecbe37f8b0a admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:27.369466 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d953ac9c-76c1-485e-98ef-5ecbe37f8b0a admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:27.374239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:27.374900 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:27.375183 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:27.375457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] PATH_INFO: `/v3/projects/0461cf8c6e9242289e67ed2fa8819749/users/0f1b24ef8c644d35a74de8c51d9aba3b/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:27.377377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0461cf8c6e9242289e67ed2fa8819749, user_id=0f1b24ef8c644d35a74de8c51d9aba3b, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:27.380128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:27.390284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1301/2621] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:27 2022] PUT /identity/v3/projects/0461cf8c6e9242289e67ed2fa8819749/users/0f1b24ef8c644d35a74de8c51d9aba3b/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:27.400402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276adf10-1c5c-4996-84c5-80a95888b162 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:27.400776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276adf10-1c5c-4996-84c5-80a95888b162 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:27.401055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276adf10-1c5c-4996-84c5-80a95888b162 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:27.429160 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-276adf10-1c5c-4996-84c5-80a95888b162 None None] MFA Rules not processed for user `0f1b24ef8c644d35a74de8c51d9aba3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:27.430592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-276adf10-1c5c-4996-84c5-80a95888b162 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:27.498352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1321/2622] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:27 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:27.509308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:27.511029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:27.570367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d5c22c-d0d3-4386-96c5-b02e3a4542f4 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:27.574040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:27.574636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:27.574918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:27.575187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:27.583924 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:27.585348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:27.594701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1302/2623] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:27 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:27.645753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276adf10-1c5c-4996-84c5-80a95888b162 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:27.647117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-276adf10-1c5c-4996-84c5-80a95888b162 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:27.659291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276adf10-1c5c-4996-84c5-80a95888b162 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:27.663521 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58cf9f5-f7b7-4999-9ffa-f9c214bde988 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:27.664109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58cf9f5-f7b7-4999-9ffa-f9c214bde988 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:27.664388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58cf9f5-f7b7-4999-9ffa-f9c214bde988 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:27.664656 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58cf9f5-f7b7-4999-9ffa-f9c214bde988 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:27.720787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58cf9f5-f7b7-4999-9ffa-f9c214bde988 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:27.722287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58cf9f5-f7b7-4999-9ffa-f9c214bde988 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:27.731894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1322/2624] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:27 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:29.701310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:29.702817 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:29.766425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb19348e-bf9a-472e-b9ca-8fa962c861fc service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:29.771573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:29.772196 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:29.772473 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:29.772741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] PATH_INFO: `/v3/users/5a1c3b4228e84dd7b5a5157368264a5f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:29.773877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a1c3b4228e84dd7b5a5157368264a5f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:29.775784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:29.947569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a1c3b4228e84dd7b5a5157368264a5f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:29.959940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a1c3b4228e84dd7b5a5157368264a5f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:29.960599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a1c3b4228e84dd7b5a5157368264a5f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:30.064641 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a1c3b4228e84dd7b5a5157368264a5f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:30.066957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:30.072830 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a1c3b4228e84dd7b5a5157368264a5f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:30.078986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1303/2625] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:29 2022] DELETE /identity/v3/users/5a1c3b4228e84dd7b5a5157368264a5f => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:30.083473 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8e55ec-c778-4549-870f-53abac5c5287 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:30.083938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8e55ec-c778-4549-870f-53abac5c5287 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:30.084054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8e55ec-c778-4549-870f-53abac5c5287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:30.102507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bf8e55ec-c778-4549-870f-53abac5c5287 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:30.103903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bf8e55ec-c778-4549-870f-53abac5c5287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:30.161094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1323/2626] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:30.170916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:30.172363 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:30.233673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aea4081-c31d-403a-a49d-addb8ce3fa4b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:30.237253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d951f2-fdf8-4170-89d0-bc9ce9ce864b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:30.237897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d951f2-fdf8-4170-89d0-bc9ce9ce864b service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:30.238175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d951f2-fdf8-4170-89d0-bc9ce9ce864b service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:30.238447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d951f2-fdf8-4170-89d0-bc9ce9ce864b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:30.247956 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d951f2-fdf8-4170-89d0-bc9ce9ce864b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:30.249385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d951f2-fdf8-4170-89d0-bc9ce9ce864b service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:30.259593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1304/2627] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:30.672141 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8e55ec-c778-4549-870f-53abac5c5287 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:30.673686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bf8e55ec-c778-4549-870f-53abac5c5287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:30.686435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8e55ec-c778-4549-870f-53abac5c5287 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:30.691266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:30.691932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:30.692234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:30.692877 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] PATH_INFO: `/v3/projects/5bfd5106918c494a962cb615d1c20b87` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:30.693707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] RBAC: Authorizing `identity:delete_project(project_id=5bfd5106918c494a962cb615d1c20b87)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:30.695616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:30.717414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:30.785468 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5bfd5106918c494a962cb615d1c20b87', 'request_id': 'req-f511871b-b73c-4d48-880e-5eb47d024ead'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:30.792139 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5bfd5106918c494a962cb615d1c20b87', 'request_id': 'req-f511871b-b73c-4d48-880e-5eb47d024ead'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:30.836602 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5bfd5106918c494a962cb615d1c20b87', 'request_id': 'req-f511871b-b73c-4d48-880e-5eb47d024ead'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:30.849911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] The token cache is being invalidate because project 5bfd5106918c494a962cb615d1c20b87 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:30.850200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:30.850834 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:30.852331 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1324/2628] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:30 2022] DELETE /identity/v3/projects/5bfd5106918c494a962cb615d1c20b87 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:30.856315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d951f2-fdf8-4170-89d0-bc9ce9ce864b service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:30.865813 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d951f2-fdf8-4170-89d0-bc9ce9ce864b service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:30.920980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:30.921626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:30.922301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:30.922301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] PATH_INFO: `/v3/users/6a4561dc99ce4323b10439693d2c54b6` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:30.923257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a4561dc99ce4323b10439693d2c54b6)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:30.925149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:31.163437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a4561dc99ce4323b10439693d2c54b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.175109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a4561dc99ce4323b10439693d2c54b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.175761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a4561dc99ce4323b10439693d2c54b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.231070 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a4561dc99ce4323b10439693d2c54b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.242585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-7e254ccb-a804-4c13-8da6-73c298932055 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a4561dc99ce4323b10439693d2c54b6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.245506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1305/2629] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:30 2022] DELETE /identity/v3/users/6a4561dc99ce4323b10439693d2c54b6 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:31.696199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:31.698055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:31.712349 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f511871b-b73c-4d48-880e-5eb47d024ead admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:31.775408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:31.776043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:31.776319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:31.776620 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] PATH_INFO: `/v3/projects/8ee1719a15cc42329d5f996f4c14aaa4` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:31.778073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ee1719a15cc42329d5f996f4c14aaa4)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:31.780085 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:31.880968 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ee1719a15cc42329d5f996f4c14aaa4', 'request_id': 'req-851de598-75f5-4cdf-895c-e2dfbd730fe1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.886518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ee1719a15cc42329d5f996f4c14aaa4', 'request_id': 'req-851de598-75f5-4cdf-895c-e2dfbd730fe1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.967933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ee1719a15cc42329d5f996f4c14aaa4', 'request_id': 'req-851de598-75f5-4cdf-895c-e2dfbd730fe1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.980971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] The token cache is being invalidate because project 8ee1719a15cc42329d5f996f4c14aaa4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:31.981260 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.981907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-851de598-75f5-4cdf-895c-e2dfbd730fe1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:31.983432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1325/2630] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:31 2022] DELETE /identity/v3/projects/8ee1719a15cc42329d5f996f4c14aaa4 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:31.988173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc936e4f-539d-47ea-810a-e3ddc47c32bf None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:31.988619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc936e4f-539d-47ea-810a-e3ddc47c32bf None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:31.988713 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc936e4f-539d-47ea-810a-e3ddc47c32bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.008835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-dc936e4f-539d-47ea-810a-e3ddc47c32bf None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:32.010076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-dc936e4f-539d-47ea-810a-e3ddc47c32bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:32.018366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:32.068420 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1306/2631] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:32.076428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baff43d-8b96-43cc-bd98-f03987c79d84 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.076714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baff43d-8b96-43cc-bd98-f03987c79d84 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.077020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baff43d-8b96-43cc-bd98-f03987c79d84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.096509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-5baff43d-8b96-43cc-bd98-f03987c79d84 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:32.097943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5baff43d-8b96-43cc-bd98-f03987c79d84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:32.106393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1326/2632] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:32.110440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc936e4f-539d-47ea-810a-e3ddc47c32bf None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:32.120787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc936e4f-539d-47ea-810a-e3ddc47c32bf None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:32.125429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:32.126064 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.126346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.126619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.127256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:24:32.127575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:32.129152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:32.141122 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1307/2633] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:24:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:24:32.145399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baff43d-8b96-43cc-bd98-f03987c79d84 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:32.154358 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baff43d-8b96-43cc-bd98-f03987c79d84 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:32.158888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d193f03-13a5-46dd-91f6-5055fc5ccd53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:32.159472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d193f03-13a5-46dd-91f6-5055fc5ccd53 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.159746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d193f03-13a5-46dd-91f6-5055fc5ccd53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.160013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d193f03-13a5-46dd-91f6-5055fc5ccd53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.160631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d193f03-13a5-46dd-91f6-5055fc5ccd53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:32.163059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d193f03-13a5-46dd-91f6-5055fc5ccd53 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:32.233378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1327/2634] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:32 2022] POST /identity/v3/projects => generated 387 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:24:32.237143 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:32.247438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b12f68c-684a-4a0b-98f5-4fe1ed626fe4 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:32.303755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7388fa-edac-43e1-b2d1-a99bf6481250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:32.304359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7388fa-edac-43e1-b2d1-a99bf6481250 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.304636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7388fa-edac-43e1-b2d1-a99bf6481250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.304905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7388fa-edac-43e1-b2d1-a99bf6481250 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.305528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7388fa-edac-43e1-b2d1-a99bf6481250 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:32.307460 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7388fa-edac-43e1-b2d1-a99bf6481250 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:32.425246 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1308/2635] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:32 2022] POST /identity/v3/users => generated 294 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:32.429336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d193f03-13a5-46dd-91f6-5055fc5ccd53 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:32.439557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d193f03-13a5-46dd-91f6-5055fc5ccd53 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:32.444059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc21c830-fbce-492a-b7ce-56b635b38f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:32.444655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc21c830-fbce-492a-b7ce-56b635b38f9e admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.444962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc21c830-fbce-492a-b7ce-56b635b38f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.445262 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc21c830-fbce-492a-b7ce-56b635b38f9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.446095 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc21c830-fbce-492a-b7ce-56b635b38f9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:32.447712 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc21c830-fbce-492a-b7ce-56b635b38f9e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:32.449842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:32.458370 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1328/2636] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:32.462303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7388fa-edac-43e1-b2d1-a99bf6481250 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:32.472366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7388fa-edac-43e1-b2d1-a99bf6481250 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:32.478104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4340652-8d57-429f-bc9f-07c82dd8ec42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:32.478642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4340652-8d57-429f-bc9f-07c82dd8ec42 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.478949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4340652-8d57-429f-bc9f-07c82dd8ec42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.479225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4340652-8d57-429f-bc9f-07c82dd8ec42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.479870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4340652-8d57-429f-bc9f-07c82dd8ec42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:32.481454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4340652-8d57-429f-bc9f-07c82dd8ec42 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:32.492242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1309/2637] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:32.496384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc21c830-fbce-492a-b7ce-56b635b38f9e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:32.506201 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc21c830-fbce-492a-b7ce-56b635b38f9e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:32.511247 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8a9b65-5b02-4ba1-90d4-c94e6486a489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:32.511889 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8a9b65-5b02-4ba1-90d4-c94e6486a489 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.512170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8a9b65-5b02-4ba1-90d4-c94e6486a489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.512448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8a9b65-5b02-4ba1-90d4-c94e6486a489 admin admin] PATH_INFO: `/v3/projects/7d861554404e4c77a3d908d1ccf5888e/users/2bdd00d4318040d5a0500c43d7ea0908/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.527343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8a9b65-5b02-4ba1-90d4-c94e6486a489 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d861554404e4c77a3d908d1ccf5888e, user_id=2bdd00d4318040d5a0500c43d7ea0908, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:32.530083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8a9b65-5b02-4ba1-90d4-c94e6486a489 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:32.608190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1329/2638] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:32 2022] PUT /identity/v3/projects/7d861554404e4c77a3d908d1ccf5888e/users/2bdd00d4318040d5a0500c43d7ea0908/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:32.619818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b459b3-4091-44a9-a472-3d98e3e0bd46 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.620194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b459b3-4091-44a9-a472-3d98e3e0bd46 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.620463 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b459b3-4091-44a9-a472-3d98e3e0bd46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.649224 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f2b459b3-4091-44a9-a472-3d98e3e0bd46 None None] MFA Rules not processed for user `2bdd00d4318040d5a0500c43d7ea0908`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:32.650447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f2b459b3-4091-44a9-a472-3d98e3e0bd46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:32.716779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1310/2639] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:32 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:32.721251 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8a9b65-5b02-4ba1-90d4-c94e6486a489 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:32.784021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8a9b65-5b02-4ba1-90d4-c94e6486a489 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:32.788504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e57eee2-8cb3-4f1c-bf4c-5ddb2b50c93c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:32.789308 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57eee2-8cb3-4f1c-bf4c-5ddb2b50c93c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.789469 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57eee2-8cb3-4f1c-bf4c-5ddb2b50c93c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.789770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e57eee2-8cb3-4f1c-bf4c-5ddb2b50c93c admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.790395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e57eee2-8cb3-4f1c-bf4c-5ddb2b50c93c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:32.792265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e57eee2-8cb3-4f1c-bf4c-5ddb2b50c93c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:32.883286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1330/2640] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:32 2022] POST /identity/v3/projects => generated 385 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:32.887564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b459b3-4091-44a9-a472-3d98e3e0bd46 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:32.899031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b459b3-4091-44a9-a472-3d98e3e0bd46 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:32.904096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:32.955522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f13ba0-13e2-4d09-816f-c4208af4c4ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:32.956137 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f13ba0-13e2-4d09-816f-c4208af4c4ad admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:32.956416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f13ba0-13e2-4d09-816f-c4208af4c4ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:32.956692 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f13ba0-13e2-4d09-816f-c4208af4c4ad admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:32.957328 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f13ba0-13e2-4d09-816f-c4208af4c4ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:32.959964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f13ba0-13e2-4d09-816f-c4208af4c4ad admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.054163 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1311/2641] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:32 2022] POST /identity/v3/users => generated 293 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:24:33.058165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e57eee2-8cb3-4f1c-bf4c-5ddb2b50c93c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.069182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e57eee2-8cb3-4f1c-bf4c-5ddb2b50c93c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:33.073818 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a2a0ef-c6ec-4791-86e8-f6fbf9d40408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:33.074408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a2a0ef-c6ec-4791-86e8-f6fbf9d40408 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.074892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a2a0ef-c6ec-4791-86e8-f6fbf9d40408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.075171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a2a0ef-c6ec-4791-86e8-f6fbf9d40408 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.075817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a2a0ef-c6ec-4791-86e8-f6fbf9d40408 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:33.077400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a2a0ef-c6ec-4791-86e8-f6fbf9d40408 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.087898 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1331/2642] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:33.091896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f13ba0-13e2-4d09-816f-c4208af4c4ad admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.101933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f13ba0-13e2-4d09-816f-c4208af4c4ad admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:33.106571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e343d0a1-bcd1-45f7-b744-446191398a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:33.107167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e343d0a1-bcd1-45f7-b744-446191398a0f admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.107444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e343d0a1-bcd1-45f7-b744-446191398a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.107729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e343d0a1-bcd1-45f7-b744-446191398a0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.108350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e343d0a1-bcd1-45f7-b744-446191398a0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:33.109977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e343d0a1-bcd1-45f7-b744-446191398a0f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.120290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1312/2643] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:33.124413 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a2a0ef-c6ec-4791-86e8-f6fbf9d40408 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.135240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a2a0ef-c6ec-4791-86e8-f6fbf9d40408 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:33.140128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b484d4e-afe6-4d2e-9fe3-612d4934b1af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:33.140764 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b484d4e-afe6-4d2e-9fe3-612d4934b1af admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.141041 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b484d4e-afe6-4d2e-9fe3-612d4934b1af admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.141315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b484d4e-afe6-4d2e-9fe3-612d4934b1af admin admin] PATH_INFO: `/v3/projects/b384b7dfd7cb4182b86e02088adaf984/users/bddf49dcc56b4ce882c31c9c272e9926/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.156294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b484d4e-afe6-4d2e-9fe3-612d4934b1af admin admin] RBAC: Authorizing `identity:create_grant(project_id=b384b7dfd7cb4182b86e02088adaf984, user_id=bddf49dcc56b4ce882c31c9c272e9926, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:33.158946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b484d4e-afe6-4d2e-9fe3-612d4934b1af admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.254269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1332/2644] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:33 2022] PUT /identity/v3/projects/b384b7dfd7cb4182b86e02088adaf984/users/bddf49dcc56b4ce882c31c9c272e9926/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:33.265085 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afc27ce-09bc-4f6d-8582-d636f0fd777a None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.265490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afc27ce-09bc-4f6d-8582-d636f0fd777a None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.265838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afc27ce-09bc-4f6d-8582-d636f0fd777a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.293053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-1afc27ce-09bc-4f6d-8582-d636f0fd777a None None] MFA Rules not processed for user `bddf49dcc56b4ce882c31c9c272e9926`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:33.294291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1afc27ce-09bc-4f6d-8582-d636f0fd777a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:33.360094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1313/2645] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:33.364985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b484d4e-afe6-4d2e-9fe3-612d4934b1af admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.429282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b484d4e-afe6-4d2e-9fe3-612d4934b1af admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:33.434153 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:33.438289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.438289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.438289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.438289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:33.438289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.543456 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1333/2646] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:33 2022] POST /identity/v3/projects => generated 385 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:33.544452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afc27ce-09bc-4f6d-8582-d636f0fd777a None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.556852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afc27ce-09bc-4f6d-8582-d636f0fd777a None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:33.592660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.594423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:33.605534 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:33.622698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e106c9f1-08b8-4db5-9ea8-e82b00eed5b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:33.623310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e106c9f1-08b8-4db5-9ea8-e82b00eed5b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.623589 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e106c9f1-08b8-4db5-9ea8-e82b00eed5b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.623858 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e106c9f1-08b8-4db5-9ea8-e82b00eed5b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.624485 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e106c9f1-08b8-4db5-9ea8-e82b00eed5b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:33.626988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e106c9f1-08b8-4db5-9ea8-e82b00eed5b5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.660175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e923aa00-4f29-41ee-be7e-cbfcfac031a7 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:33.665843 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:33.666481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.666759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.667031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] PATH_INFO: `/v3/users/40e345e2deec435a953b99f01c7049c1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.668161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=40e345e2deec435a953b99f01c7049c1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:33.670123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.756585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1314/2647] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:33 2022] POST /identity/v3/users => generated 299 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:24:33.760364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e106c9f1-08b8-4db5-9ea8-e82b00eed5b5 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.769657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e106c9f1-08b8-4db5-9ea8-e82b00eed5b5 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:33.774745 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a61157-3690-42af-80a1-d1d74217202d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:33.775316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a61157-3690-42af-80a1-d1d74217202d admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.775595 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a61157-3690-42af-80a1-d1d74217202d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.775865 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a61157-3690-42af-80a1-d1d74217202d admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.776484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a61157-3690-42af-80a1-d1d74217202d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:33.778136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a61157-3690-42af-80a1-d1d74217202d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.779370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:33.790535 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1315/2648] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:24:33.796838 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a61157-3690-42af-80a1-d1d74217202d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.806735 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a61157-3690-42af-80a1-d1d74217202d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:33.811325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:33.812045 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.812256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.812528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] PATH_INFO: `/v3/projects/84a0d7fcafb849f0a8b0549595148ce9/users/657a21216a7542e593dcbcb5a60ee566/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:33.827242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] RBAC: Authorizing `identity:create_grant(project_id=84a0d7fcafb849f0a8b0549595148ce9, user_id=657a21216a7542e593dcbcb5a60ee566, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:33.829886 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:33.877772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40e345e2deec435a953b99f01c7049c1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:33.878414 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40e345e2deec435a953b99f01c7049c1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:33.890067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40e345e2deec435a953b99f01c7049c1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:33.974947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40e345e2deec435a953b99f01c7049c1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:33.981436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1316/2649] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:33 2022] PUT /identity/v3/projects/84a0d7fcafb849f0a8b0549595148ce9/users/657a21216a7542e593dcbcb5a60ee566/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:33.984618 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5106bdb6-9135-4dff-8d3b-8c7249aa15dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40e345e2deec435a953b99f01c7049c1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:33.985415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:33.987073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:33.990217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1334/2650] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:33 2022] DELETE /identity/v3/users/40e345e2deec435a953b99f01c7049c1 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:24:33.994409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4b5dd-12bf-4741-91a8-4f3a3e20ed25 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:33.994970 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4b5dd-12bf-4741-91a8-4f3a3e20ed25 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:33.995248 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4b5dd-12bf-4741-91a8-4f3a3e20ed25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:34.014679 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-ffd4b5dd-12bf-4741-91a8-4f3a3e20ed25 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:34.016009 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ffd4b5dd-12bf-4741-91a8-4f3a3e20ed25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:34.050006 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f823dd8-c513-4cb6-a7f8-9852e8b5806d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:34.074790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1335/2651] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:24:34.085126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4b5dd-12bf-4741-91a8-4f3a3e20ed25 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:34.086663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ffd4b5dd-12bf-4741-91a8-4f3a3e20ed25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:34.109172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f405c62f-6e4c-463b-b85b-bd116f3e9399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:34.109947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f405c62f-6e4c-463b-b85b-bd116f3e9399 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:34.110230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f405c62f-6e4c-463b-b85b-bd116f3e9399 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:34.110502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f405c62f-6e4c-463b-b85b-bd116f3e9399 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:34.111139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f405c62f-6e4c-463b-b85b-bd116f3e9399 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:34.112719 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f405c62f-6e4c-463b-b85b-bd116f3e9399 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:34.122587 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1317/2652] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:33 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:24:34.126764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f405c62f-6e4c-463b-b85b-bd116f3e9399 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:34.137603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f405c62f-6e4c-463b-b85b-bd116f3e9399 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:34.142331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addef810-9924-427b-8437-14f72bfd91be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:34.142973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addef810-9924-427b-8437-14f72bfd91be admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:34.143244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4b5dd-12bf-4741-91a8-4f3a3e20ed25 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:34.143352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addef810-9924-427b-8437-14f72bfd91be admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:34.143629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addef810-9924-427b-8437-14f72bfd91be admin admin] PATH_INFO: `/v3/projects/84a0d7fcafb849f0a8b0549595148ce9/users/657a21216a7542e593dcbcb5a60ee566/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:34.146121 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addef810-9924-427b-8437-14f72bfd91be admin admin] RBAC: Authorizing `identity:create_grant(project_id=84a0d7fcafb849f0a8b0549595148ce9, user_id=657a21216a7542e593dcbcb5a60ee566, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:34.147476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:34.148078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:34.148359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:34.148633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:34.148731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addef810-9924-427b-8437-14f72bfd91be admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:34.157974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:34.159234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1318/2653] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:34 2022] PUT /identity/v3/projects/84a0d7fcafb849f0a8b0549595148ce9/users/657a21216a7542e593dcbcb5a60ee566/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:24:34.159528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:34.169126 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2ca308-d29a-45a3-bf04-c5bd7c39db87 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:34.169413 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2ca308-d29a-45a3-bf04-c5bd7c39db87 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:34.169522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1336/2654] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:24:34.169778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2ca308-d29a-45a3-bf04-c5bd7c39db87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:34.198788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ef2ca308-d29a-45a3-bf04-c5bd7c39db87 None None] MFA Rules not processed for user `657a21216a7542e593dcbcb5a60ee566`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:34.200037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ef2ca308-d29a-45a3-bf04-c5bd7c39db87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:34.274211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1319/2655] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:34 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:24:34.284911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:34.286392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:34.342266 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b316dd18-d68a-4b93-8ba9-faca0d675f5b service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:34.405258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641699bf-9a0e-4299-acb2-558c1c1725ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:34.407650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641699bf-9a0e-4299-acb2-558c1c1725ae service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:34.407650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641699bf-9a0e-4299-acb2-558c1c1725ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:34.407650 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641699bf-9a0e-4299-acb2-558c1c1725ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:34.419925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641699bf-9a0e-4299-acb2-558c1c1725ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:34.421558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641699bf-9a0e-4299-acb2-558c1c1725ae service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:34.430198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:34.435958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1337/2656] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:34 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:34.516727 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:34.637864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2ca308-d29a-45a3-bf04-c5bd7c39db87 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:34.639369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ef2ca308-d29a-45a3-bf04-c5bd7c39db87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:34.722367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2ca308-d29a-45a3-bf04-c5bd7c39db87 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:34.727282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:34.727939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:34.728217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:34.728487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] PATH_INFO: `/v3/projects/ef24ba5d0a144b688312fb238cf0ebe6` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:34.729655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef24ba5d0a144b688312fb238cf0ebe6)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:34.731558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:34.847869 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef24ba5d0a144b688312fb238cf0ebe6', 'request_id': 'req-66ddf65c-c2ab-4063-b184-c07279096641'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:34.944837 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef24ba5d0a144b688312fb238cf0ebe6', 'request_id': 'req-66ddf65c-c2ab-4063-b184-c07279096641'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:34.950811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef24ba5d0a144b688312fb238cf0ebe6', 'request_id': 'req-66ddf65c-c2ab-4063-b184-c07279096641'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:34.961893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] The token cache is being invalidate because project ef24ba5d0a144b688312fb238cf0ebe6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:34.962182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:34.962829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:34.964307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1320/2657] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:34 2022] DELETE /identity/v3/projects/ef24ba5d0a144b688312fb238cf0ebe6 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:34.968492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641699bf-9a0e-4299-acb2-558c1c1725ae service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:34.979424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641699bf-9a0e-4299-acb2-558c1c1725ae service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:35.068888 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:35.069513 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:35.069817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:35.070090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] PATH_INFO: `/v3/users/0f1b24ef8c644d35a74de8c51d9aba3b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:35.071192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f1b24ef8c644d35a74de8c51d9aba3b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:35.073091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:35.284775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:35.297455 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f1b24ef8c644d35a74de8c51d9aba3b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:35.298138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f1b24ef8c644d35a74de8c51d9aba3b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:35.310540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f1b24ef8c644d35a74de8c51d9aba3b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:35.401921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f1b24ef8c644d35a74de8c51d9aba3b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:35.410517 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f1b24ef8c644d35a74de8c51d9aba3b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:35.417692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1338/2658] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:34 2022] DELETE /identity/v3/users/0f1b24ef8c644d35a74de8c51d9aba3b => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:35.595646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:35.597075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:35.656490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ddf65c-c2ab-4063-b184-c07279096641 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:35.691620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:35.711695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d377bcf-b04e-4c18-a749-9ab245950ff8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:35.712310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d377bcf-b04e-4c18-a749-9ab245950ff8 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:35.712591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d377bcf-b04e-4c18-a749-9ab245950ff8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:35.712867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d377bcf-b04e-4c18-a749-9ab245950ff8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:35.714662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8d377bcf-b04e-4c18-a749-9ab245950ff8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:35.779471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d377bcf-b04e-4c18-a749-9ab245950ff8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:35.780898 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d377bcf-b04e-4c18-a749-9ab245950ff8 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:35.789696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1321/2659] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:35 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:35.916868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:35.918338 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:35.985199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0af1966-bac8-4def-bf81-9210560a8408 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:35.990451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:35.991082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:35.991357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:35.991630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] PATH_INFO: `/v3/projects/0461cf8c6e9242289e67ed2fa8819749` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:35.992759 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=0461cf8c6e9242289e67ed2fa8819749)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:35.994792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:36.266819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0461cf8c6e9242289e67ed2fa8819749', 'request_id': 'req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:36.272289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0461cf8c6e9242289e67ed2fa8819749', 'request_id': 'req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:36.367398 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0461cf8c6e9242289e67ed2fa8819749', 'request_id': 'req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:36.379035 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] The token cache is being invalidate because project 0461cf8c6e9242289e67ed2fa8819749 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:36.379345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:36.379963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-a0e52bad-fcc7-46f2-b4f2-d67eac87af3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:36.381442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1339/2660] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:35 2022] DELETE /identity/v3/projects/0461cf8c6e9242289e67ed2fa8819749 => generated 0 bytes in 465 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:24:36.386451 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5341063f-a115-42ca-a02d-c7577fb941e6 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:36.386770 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5341063f-a115-42ca-a02d-c7577fb941e6 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:36.387047 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5341063f-a115-42ca-a02d-c7577fb941e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:36.406867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5341063f-a115-42ca-a02d-c7577fb941e6 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:36.408072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5341063f-a115-42ca-a02d-c7577fb941e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:36.469681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1322/2661] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:36.477990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a2360d-a309-4785-800e-426fb87652d4 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:36.478284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a2360d-a309-4785-800e-426fb87652d4 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:36.478562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a2360d-a309-4785-800e-426fb87652d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:36.499391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-08a2360d-a309-4785-800e-426fb87652d4 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:36.500603 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-08a2360d-a309-4785-800e-426fb87652d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:36.509194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1340/2662] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:36.513804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5341063f-a115-42ca-a02d-c7577fb941e6 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:36.523739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5341063f-a115-42ca-a02d-c7577fb941e6 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:36.532157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:36.535270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:36.535270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:36.535270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:36.535270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:24:36.535270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:36.536404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:36.552227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1323/2663] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:24:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:24:36.553339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a2360d-a309-4785-800e-426fb87652d4 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:36.562899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a2360d-a309-4785-800e-426fb87652d4 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:36.567803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935e7a42-7e79-4604-9a9c-df108fc11efa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:36.568409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935e7a42-7e79-4604-9a9c-df108fc11efa admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:36.568688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935e7a42-7e79-4604-9a9c-df108fc11efa admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:36.568960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935e7a42-7e79-4604-9a9c-df108fc11efa admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:36.569599 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935e7a42-7e79-4604-9a9c-df108fc11efa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:36.571582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935e7a42-7e79-4604-9a9c-df108fc11efa admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:36.670986 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1341/2664] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:36 2022] POST /identity/v3/projects => generated 371 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:36.675080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:36.685829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a65e711-90a8-4596-8a54-1d9e91465d7b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:36.743225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d90e0-34d9-4fe5-b1f6-6a84b37f7f34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:36.743836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d90e0-34d9-4fe5-b1f6-6a84b37f7f34 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:36.744116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d90e0-34d9-4fe5-b1f6-6a84b37f7f34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:36.744391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753d90e0-34d9-4fe5-b1f6-6a84b37f7f34 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:36.745018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d90e0-34d9-4fe5-b1f6-6a84b37f7f34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:36.746974 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753d90e0-34d9-4fe5-b1f6-6a84b37f7f34 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:36.813842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1324/2665] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:36 2022] POST /identity/v3/users => generated 286 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:36.817952 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935e7a42-7e79-4604-9a9c-df108fc11efa admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:36.828100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935e7a42-7e79-4604-9a9c-df108fc11efa admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:36.833254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b4a3eb-9c70-41a5-b5ae-7e3d88d2fbdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:36.833903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b4a3eb-9c70-41a5-b5ae-7e3d88d2fbdb admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:36.834182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b4a3eb-9c70-41a5-b5ae-7e3d88d2fbdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:36.834457 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b4a3eb-9c70-41a5-b5ae-7e3d88d2fbdb admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:36.838075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b4a3eb-9c70-41a5-b5ae-7e3d88d2fbdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:36.838075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b4a3eb-9c70-41a5-b5ae-7e3d88d2fbdb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:36.842242 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:36.847365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1342/2666] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:36.852156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d90e0-34d9-4fe5-b1f6-6a84b37f7f34 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:36.858441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:36.862176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753d90e0-34d9-4fe5-b1f6-6a84b37f7f34 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:36.866849 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3c3bf0-b29b-4fee-82ce-8d5535a169b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:36.867464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3c3bf0-b29b-4fee-82ce-8d5535a169b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:36.867747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3c3bf0-b29b-4fee-82ce-8d5535a169b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:36.868017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3c3bf0-b29b-4fee-82ce-8d5535a169b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:36.868645 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3c3bf0-b29b-4fee-82ce-8d5535a169b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:36.870305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3c3bf0-b29b-4fee-82ce-8d5535a169b6 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:36.880478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1325/2667] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:36.884502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b4a3eb-9c70-41a5-b5ae-7e3d88d2fbdb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:36.894494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b4a3eb-9c70-41a5-b5ae-7e3d88d2fbdb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:36.899644 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:36.900293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:36.900572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:36.900845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] PATH_INFO: `/v3/projects/684e3adc0a9148cda29de351c4f066fe/users/3d598ffecf5744ccb038e088f039f037/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:36.916124 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=684e3adc0a9148cda29de351c4f066fe, user_id=3d598ffecf5744ccb038e088f039f037, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:36.918797 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:37.007901 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1343/2668] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:36 2022] PUT /identity/v3/projects/684e3adc0a9148cda29de351c4f066fe/users/3d598ffecf5744ccb038e088f039f037/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:24:37.013140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd92434-74a7-416a-832e-9d7e3c92ca62 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:37.013431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd92434-74a7-416a-832e-9d7e3c92ca62 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:37.013755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd92434-74a7-416a-832e-9d7e3c92ca62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:37.032909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-dcd92434-74a7-416a-832e-9d7e3c92ca62 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:37.034147 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-dcd92434-74a7-416a-832e-9d7e3c92ca62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:37.093177 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1326/2669] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:37.102829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:37.104555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:37.161815 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026bc76-566a-4fd4-b5ae-5745d4c900ff admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:37.165467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5589738a-1da2-4750-87ba-3272944972c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:37.166138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5589738a-1da2-4750-87ba-3272944972c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:37.166419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5589738a-1da2-4750-87ba-3272944972c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:37.166695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5589738a-1da2-4750-87ba-3272944972c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:37.175969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5589738a-1da2-4750-87ba-3272944972c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:37.177421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5589738a-1da2-4750-87ba-3272944972c4 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:37.186899 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1344/2670] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:38.441970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c1b86e-dac3-41fa-91b7-d5241a868d30 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:38.442435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c1b86e-dac3-41fa-91b7-d5241a868d30 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:38.442540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c1b86e-dac3-41fa-91b7-d5241a868d30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:38.470640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-40c1b86e-dac3-41fa-91b7-d5241a868d30 None None] MFA Rules not processed for user `3d598ffecf5744ccb038e088f039f037`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:38.471861 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-40c1b86e-dac3-41fa-91b7-d5241a868d30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:38.542271 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1327/2671] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:38 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:38.546975 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5589738a-1da2-4750-87ba-3272944972c4 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:38.556451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5589738a-1da2-4750-87ba-3272944972c4 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:38.561032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7634e009-e7e8-4de8-bacb-7e422f3a05eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:38.561640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7634e009-e7e8-4de8-bacb-7e422f3a05eb admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:38.561923 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7634e009-e7e8-4de8-bacb-7e422f3a05eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:38.562190 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7634e009-e7e8-4de8-bacb-7e422f3a05eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:38.562807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7634e009-e7e8-4de8-bacb-7e422f3a05eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:38.564690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7634e009-e7e8-4de8-bacb-7e422f3a05eb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:38.625600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1345/2672] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:38 2022] POST /identity/v3/projects => generated 371 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:38.630100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c1b86e-dac3-41fa-91b7-d5241a868d30 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:38.640540 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c1b86e-dac3-41fa-91b7-d5241a868d30 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:38.701030 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630dda2-887f-4928-a057-09c48cfa3fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:38.701688 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630dda2-887f-4928-a057-09c48cfa3fbe admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:38.701969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630dda2-887f-4928-a057-09c48cfa3fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:38.702244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6630dda2-887f-4928-a057-09c48cfa3fbe admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:38.702862 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6630dda2-887f-4928-a057-09c48cfa3fbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:38.704771 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6630dda2-887f-4928-a057-09c48cfa3fbe admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:38.802072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1328/2673] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:38 2022] POST /identity/v3/users => generated 292 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:38.806048 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7634e009-e7e8-4de8-bacb-7e422f3a05eb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:38.816522 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7634e009-e7e8-4de8-bacb-7e422f3a05eb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:38.821630 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8166ff82-ce39-4c05-89e4-cccb164e8893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:38.822220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8166ff82-ce39-4c05-89e4-cccb164e8893 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:38.822496 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8166ff82-ce39-4c05-89e4-cccb164e8893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:38.822766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8166ff82-ce39-4c05-89e4-cccb164e8893 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:38.823399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8166ff82-ce39-4c05-89e4-cccb164e8893 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:38.824965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8166ff82-ce39-4c05-89e4-cccb164e8893 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:38.835984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1346/2674] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:38.839909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630dda2-887f-4928-a057-09c48cfa3fbe admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:38.850034 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6630dda2-887f-4928-a057-09c48cfa3fbe admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:38.854664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4ce1b1-b616-4ecb-83d3-767492cad533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:38.855315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4ce1b1-b616-4ecb-83d3-767492cad533 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:38.855596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4ce1b1-b616-4ecb-83d3-767492cad533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:38.855872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4ce1b1-b616-4ecb-83d3-767492cad533 admin admin] PATH_INFO: `/v3/projects/064e370f28e8496a9ab3f6e93a3ef03f/users/14354f56ed6f4ec8a9273c9c80d2faee/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:38.866864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:38.870711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4ce1b1-b616-4ecb-83d3-767492cad533 admin admin] RBAC: Authorizing `identity:create_grant(project_id=064e370f28e8496a9ab3f6e93a3ef03f, user_id=14354f56ed6f4ec8a9273c9c80d2faee, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:38.873362 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4ce1b1-b616-4ecb-83d3-767492cad533 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:38.929025 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1329/2675] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:38 2022] PUT /identity/v3/projects/064e370f28e8496a9ab3f6e93a3ef03f/users/14354f56ed6f4ec8a9273c9c80d2faee/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:38.934601 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8166ff82-ce39-4c05-89e4-cccb164e8893 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:38.945192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8166ff82-ce39-4c05-89e4-cccb164e8893 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:39.001407 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b4bc09-0583-4fbd-a211-107431e83280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:39.002050 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4bc09-0583-4fbd-a211-107431e83280 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:39.003845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4bc09-0583-4fbd-a211-107431e83280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:39.003845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b4bc09-0583-4fbd-a211-107431e83280 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:39.003845 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b4bc09-0583-4fbd-a211-107431e83280 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:39.004893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b4bc09-0583-4fbd-a211-107431e83280 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:39.015575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1347/2676] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:38 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:39.019707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4ce1b1-b616-4ecb-83d3-767492cad533 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:39.030203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4ce1b1-b616-4ecb-83d3-767492cad533 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:39.035069 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:39.035821 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:39.038188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:39.038188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] PATH_INFO: `/v3/projects/064e370f28e8496a9ab3f6e93a3ef03f/users/14354f56ed6f4ec8a9273c9c80d2faee/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:39.038381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] RBAC: Authorizing `identity:create_grant(project_id=064e370f28e8496a9ab3f6e93a3ef03f, user_id=14354f56ed6f4ec8a9273c9c80d2faee, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:39.041010 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:39.051490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1330/2677] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:39 2022] PUT /identity/v3/projects/064e370f28e8496a9ab3f6e93a3ef03f/users/14354f56ed6f4ec8a9273c9c80d2faee/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:40.353126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97bdc4-9d5d-4431-aea0-2d4261e64318 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:40.353126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97bdc4-9d5d-4431-aea0-2d4261e64318 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:40.353126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af97bdc4-9d5d-4431-aea0-2d4261e64318 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:40.374807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-af97bdc4-9d5d-4431-aea0-2d4261e64318 None None] MFA Rules not processed for user `14354f56ed6f4ec8a9273c9c80d2faee`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:40.376012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-af97bdc4-9d5d-4431-aea0-2d4261e64318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:40.408745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:40.449040 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1348/2678] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:40 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:40.460583 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:40.462314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:40.521612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5be675-99d1-42ce-bb1f-91f1cf9d664c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:40.525232 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6039d33-fd3b-4789-a1ba-aa83cd2f75a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:40.525856 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6039d33-fd3b-4789-a1ba-aa83cd2f75a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:40.526132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6039d33-fd3b-4789-a1ba-aa83cd2f75a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:40.526400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6039d33-fd3b-4789-a1ba-aa83cd2f75a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:40.584214 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6039d33-fd3b-4789-a1ba-aa83cd2f75a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:40.585846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6039d33-fd3b-4789-a1ba-aa83cd2f75a7 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:40.595187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1331/2679] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:40 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:40.874732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af97bdc4-9d5d-4431-aea0-2d4261e64318 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:40.876147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-af97bdc4-9d5d-4431-aea0-2d4261e64318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:40.888084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af97bdc4-9d5d-4431-aea0-2d4261e64318 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:40.891681 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:40.892324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:40.892647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:40.892947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:40.894690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:40.953962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:40.955385 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:40.964205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1349/2680] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:40 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:43.742205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6039d33-fd3b-4789-a1ba-aa83cd2f75a7 service cinder] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:43.750675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6039d33-fd3b-4789-a1ba-aa83cd2f75a7 service cinder] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:43.754208 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8590d4e-d9dd-4bc7-bc3f-6c9a80f07bf3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:43.754799 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8590d4e-d9dd-4bc7-bc3f-6c9a80f07bf3 service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:43.755076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8590d4e-d9dd-4bc7-bc3f-6c9a80f07bf3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:43.755345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8590d4e-d9dd-4bc7-bc3f-6c9a80f07bf3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:43.764698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8590d4e-d9dd-4bc7-bc3f-6c9a80f07bf3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:43.766152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8590d4e-d9dd-4bc7-bc3f-6c9a80f07bf3 service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:43.775318 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1332/2681] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:43 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:49.803212 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:49.805239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:49.869862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f057ff-cfaf-4fa0-99cd-894b36e4d3a2 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:49.874526 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:49.875149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:49.875431 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:49.875702 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] PATH_INFO: `/v3/users/2bdd00d4318040d5a0500c43d7ea0908` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:49.876813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bdd00d4318040d5a0500c43d7ea0908)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:49.878745 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:50.083373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bdd00d4318040d5a0500c43d7ea0908'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:50.083937 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bdd00d4318040d5a0500c43d7ea0908'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:50.095848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bdd00d4318040d5a0500c43d7ea0908'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:50.098707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:50.218600 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bdd00d4318040d5a0500c43d7ea0908'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:50.226302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bdd00d4318040d5a0500c43d7ea0908'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:50.234122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1350/2682] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:49 2022] DELETE /identity/v3/users/2bdd00d4318040d5a0500c43d7ea0908 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:50.238737 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa67c2-4eb2-4635-bca7-1464451effc9 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:50.239244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa67c2-4eb2-4635-bca7-1464451effc9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:50.239343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa67c2-4eb2-4635-bca7-1464451effc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:50.258029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-deaa67c2-4eb2-4635-bca7-1464451effc9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:50.259223 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-deaa67c2-4eb2-4635-bca7-1464451effc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:50.316969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1333/2683] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:50.326673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:50.328081 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:50.385858 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4a5d1d-a361-49c0-a52b-7b9eaf9acf2d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:50.389485 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9103b5e6-95ed-437b-b0a4-42b23d23fe8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:50.394099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9103b5e6-95ed-437b-b0a4-42b23d23fe8b service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:50.394099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9103b5e6-95ed-437b-b0a4-42b23d23fe8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:50.394099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9103b5e6-95ed-437b-b0a4-42b23d23fe8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:50.400504 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9103b5e6-95ed-437b-b0a4-42b23d23fe8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:50.402020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9103b5e6-95ed-437b-b0a4-42b23d23fe8b service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:50.411327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1351/2684] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:24:50.896049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaa67c2-4eb2-4635-bca7-1464451effc9 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:50.901662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-deaa67c2-4eb2-4635-bca7-1464451effc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:50.915952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaa67c2-4eb2-4635-bca7-1464451effc9 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:50.921707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:50.922345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:50.922651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:50.922880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] PATH_INFO: `/v3/projects/7d861554404e4c77a3d908d1ccf5888e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:50.924162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d861554404e4c77a3d908d1ccf5888e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:50.930603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:50.938430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:51.088296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d861554404e4c77a3d908d1ccf5888e', 'request_id': 'req-9c108eae-1e2b-428a-80bc-65ad635a9d1e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.177452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d861554404e4c77a3d908d1ccf5888e', 'request_id': 'req-9c108eae-1e2b-428a-80bc-65ad635a9d1e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.184829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d861554404e4c77a3d908d1ccf5888e', 'request_id': 'req-9c108eae-1e2b-428a-80bc-65ad635a9d1e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.198013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] The token cache is being invalidate because project 7d861554404e4c77a3d908d1ccf5888e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:51.198314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.199435 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.201187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1334/2685] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:50 2022] DELETE /identity/v3/projects/7d861554404e4c77a3d908d1ccf5888e => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:51.204956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9103b5e6-95ed-437b-b0a4-42b23d23fe8b service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:51.214776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9103b5e6-95ed-437b-b0a4-42b23d23fe8b service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:51.281570 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:51.282227 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:51.282511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:51.282787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] PATH_INFO: `/v3/users/bddf49dcc56b4ce882c31c9c272e9926` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:51.283913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] RBAC: Authorizing `identity:delete_user(user_id=bddf49dcc56b4ce882c31c9c272e9926)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:51.285966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:51.490648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bddf49dcc56b4ce882c31c9c272e9926'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.491392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bddf49dcc56b4ce882c31c9c272e9926'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.504890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bddf49dcc56b4ce882c31c9c272e9926'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.577100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bddf49dcc56b4ce882c31c9c272e9926'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.584422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bddf49dcc56b4ce882c31c9c272e9926'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:51.590341 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1352/2686] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:51 2022] DELETE /identity/v3/users/bddf49dcc56b4ce882c31c9c272e9926 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:52.003352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:52.004823 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:52.019904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c108eae-1e2b-428a-80bc-65ad635a9d1e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:52.078049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:52.078698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:52.078963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:52.079218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] PATH_INFO: `/v3/projects/b384b7dfd7cb4182b86e02088adaf984` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:52.080333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b384b7dfd7cb4182b86e02088adaf984)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:52.082261 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:52.167587 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b384b7dfd7cb4182b86e02088adaf984', 'request_id': 'req-35e39de2-812c-4616-b493-af858e0fefc9'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.275293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b384b7dfd7cb4182b86e02088adaf984', 'request_id': 'req-35e39de2-812c-4616-b493-af858e0fefc9'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.281273 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b384b7dfd7cb4182b86e02088adaf984', 'request_id': 'req-35e39de2-812c-4616-b493-af858e0fefc9'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.292351 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] The token cache is being invalidate because project b384b7dfd7cb4182b86e02088adaf984 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:52.292638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.293277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.294768 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1335/2687] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:52 2022] DELETE /identity/v3/projects/b384b7dfd7cb4182b86e02088adaf984 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:52.298716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:52.300216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:52.312664 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec49e0-88ce-4ea5-95ff-ac1189d3d58e admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:52.316380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:52.369994 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:52.370615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:52.370892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:52.371164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] PATH_INFO: `/v3/users/657a21216a7542e593dcbcb5a60ee566` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:52.372259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] RBAC: Authorizing `identity:delete_user(user_id=657a21216a7542e593dcbcb5a60ee566)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:52.374193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:52.569890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '657a21216a7542e593dcbcb5a60ee566'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.571224 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '657a21216a7542e593dcbcb5a60ee566'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.583386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '657a21216a7542e593dcbcb5a60ee566'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.669918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '657a21216a7542e593dcbcb5a60ee566'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.677514 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-b96f56b1-48f1-450f-8b9e-f2781530488a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '657a21216a7542e593dcbcb5a60ee566'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:52.683983 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1353/2688] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:24:52 2022] DELETE /identity/v3/users/657a21216a7542e593dcbcb5a60ee566 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:53.147073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:53.155588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:53.171154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e39de2-812c-4616-b493-af858e0fefc9 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:53.201073 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:53.229714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:53.230640 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:53.231205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:53.232014 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] PATH_INFO: `/v3/projects/84a0d7fcafb849f0a8b0549595148ce9` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:53.233198 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=84a0d7fcafb849f0a8b0549595148ce9)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:53.235178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:53.361465 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84a0d7fcafb849f0a8b0549595148ce9', 'request_id': 'req-251af677-4263-461e-8f8e-a0d4e6339b3f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:53.452687 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84a0d7fcafb849f0a8b0549595148ce9', 'request_id': 'req-251af677-4263-461e-8f8e-a0d4e6339b3f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:53.458541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84a0d7fcafb849f0a8b0549595148ce9', 'request_id': 'req-251af677-4263-461e-8f8e-a0d4e6339b3f'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:53.470664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] The token cache is being invalidate because project 84a0d7fcafb849f0a8b0549595148ce9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:24:53.470949 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:53.471570 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-251af677-4263-461e-8f8e-a0d4e6339b3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:24:53.473060 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1336/2689] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:24:53 2022] DELETE /identity/v3/projects/84a0d7fcafb849f0a8b0549595148ce9 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:53.480053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f9cb43-5ecc-48c5-b518-d8dee7f93f2c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:53.480605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f9cb43-5ecc-48c5-b518-d8dee7f93f2c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:53.480605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f9cb43-5ecc-48c5-b518-d8dee7f93f2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:53.499790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-75f9cb43-5ecc-48c5-b518-d8dee7f93f2c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:53.500984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-75f9cb43-5ecc-48c5-b518-d8dee7f93f2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:53.560940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1354/2690] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:53.569444 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c7f321-8a23-422b-92ef-ee11fbc2f78b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:53.569763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c7f321-8a23-422b-92ef-ee11fbc2f78b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:53.570593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c7f321-8a23-422b-92ef-ee11fbc2f78b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:53.590562 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-93c7f321-8a23-422b-92ef-ee11fbc2f78b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:53.591769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-93c7f321-8a23-422b-92ef-ee11fbc2f78b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:53.599863 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1337/2691] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:24:53.604234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f9cb43-5ecc-48c5-b518-d8dee7f93f2c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:53.613963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f9cb43-5ecc-48c5-b518-d8dee7f93f2c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:53.618802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:53.619429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:53.619711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:53.620008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:53.620661 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:24:53.620963 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:53.622592 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:53.634929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1355/2692] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:24:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:24:53.639103 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c7f321-8a23-422b-92ef-ee11fbc2f78b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:53.649350 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c7f321-8a23-422b-92ef-ee11fbc2f78b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:53.654109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfb5c55-faff-4a55-bf01-9c12c74fd714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:53.654705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfb5c55-faff-4a55-bf01-9c12c74fd714 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:53.654985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfb5c55-faff-4a55-bf01-9c12c74fd714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:53.655256 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfb5c55-faff-4a55-bf01-9c12c74fd714 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:53.655927 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfb5c55-faff-4a55-bf01-9c12c74fd714 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:53.657902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfb5c55-faff-4a55-bf01-9c12c74fd714 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:53.757235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1338/2693] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:53 2022] POST /identity/v3/projects => generated 377 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:53.762086 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:53.772240 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1996998c-8a90-42b5-9b51-090e4f395dad admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:53.781645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:53.828442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94564abf-cff3-4d2e-adf5-692c2852c368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:53.829037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94564abf-cff3-4d2e-adf5-692c2852c368 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:53.829312 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94564abf-cff3-4d2e-adf5-692c2852c368 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:53.829580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94564abf-cff3-4d2e-adf5-692c2852c368 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:53.830216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94564abf-cff3-4d2e-adf5-692c2852c368 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:53.832138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94564abf-cff3-4d2e-adf5-692c2852c368 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:53.938855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1356/2694] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:53 2022] POST /identity/v3/users => generated 289 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:53.943254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfb5c55-faff-4a55-bf01-9c12c74fd714 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:53.954207 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfb5c55-faff-4a55-bf01-9c12c74fd714 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:53.958836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643a1cee-b408-4b22-a584-caf9954bc069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:53.959442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643a1cee-b408-4b22-a584-caf9954bc069 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:53.959723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643a1cee-b408-4b22-a584-caf9954bc069 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:53.960022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643a1cee-b408-4b22-a584-caf9954bc069 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:53.960763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643a1cee-b408-4b22-a584-caf9954bc069 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:53.962410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643a1cee-b408-4b22-a584-caf9954bc069 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:53.972475 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1339/2695] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:24:53.977842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94564abf-cff3-4d2e-adf5-692c2852c368 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:53.987744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94564abf-cff3-4d2e-adf5-692c2852c368 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:53.993179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639c855f-4871-446f-82c1-ebcb1673c9bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:53.993846 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639c855f-4871-446f-82c1-ebcb1673c9bf admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:53.994125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639c855f-4871-446f-82c1-ebcb1673c9bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:53.994446 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639c855f-4871-446f-82c1-ebcb1673c9bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:53.995070 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639c855f-4871-446f-82c1-ebcb1673c9bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:53.996646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639c855f-4871-446f-82c1-ebcb1673c9bf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:54.007010 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1357/2696] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:54.011760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643a1cee-b408-4b22-a584-caf9954bc069 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:54.021577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643a1cee-b408-4b22-a584-caf9954bc069 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:54.026240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89457674-b8bd-4b35-bd7a-94587640a742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:54.026883 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89457674-b8bd-4b35-bd7a-94587640a742 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.027160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89457674-b8bd-4b35-bd7a-94587640a742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.027434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89457674-b8bd-4b35-bd7a-94587640a742 admin admin] PATH_INFO: `/v3/projects/43de1928dee44836bcc5388a8a27cd9d/users/6b411d8340e249ae806f49ce8c4056d6/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.041677 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89457674-b8bd-4b35-bd7a-94587640a742 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43de1928dee44836bcc5388a8a27cd9d, user_id=6b411d8340e249ae806f49ce8c4056d6, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:54.044315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89457674-b8bd-4b35-bd7a-94587640a742 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:54.115811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1340/2697] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:54 2022] PUT /identity/v3/projects/43de1928dee44836bcc5388a8a27cd9d/users/6b411d8340e249ae806f49ce8c4056d6/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:54.127488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74b3606-6f33-4f60-956d-20ce3350172c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.127827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74b3606-6f33-4f60-956d-20ce3350172c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.128101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74b3606-6f33-4f60-956d-20ce3350172c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.155832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c74b3606-6f33-4f60-956d-20ce3350172c None None] MFA Rules not processed for user `6b411d8340e249ae806f49ce8c4056d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:54.157023 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c74b3606-6f33-4f60-956d-20ce3350172c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:54.226191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1358/2698] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:24:54.231276 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89457674-b8bd-4b35-bd7a-94587640a742 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:54.288490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:54.292678 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89457674-b8bd-4b35-bd7a-94587640a742 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:54.297607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e1de50-c6d1-41e5-af7c-4ee15ebbd1ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:54.298234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e1de50-c6d1-41e5-af7c-4ee15ebbd1ad admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.298511 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e1de50-c6d1-41e5-af7c-4ee15ebbd1ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.298781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e1de50-c6d1-41e5-af7c-4ee15ebbd1ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.299430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e1de50-c6d1-41e5-af7c-4ee15ebbd1ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:54.301316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e1de50-c6d1-41e5-af7c-4ee15ebbd1ad admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:54.409735 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1341/2699] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:54 2022] POST /identity/v3/projects => generated 379 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:24:54.414444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74b3606-6f33-4f60-956d-20ce3350172c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:54.425186 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74b3606-6f33-4f60-956d-20ce3350172c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:54.481384 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae8d1c-050b-4cac-b454-b67762c484bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:54.482076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae8d1c-050b-4cac-b454-b67762c484bc admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.482289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae8d1c-050b-4cac-b454-b67762c484bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.482558 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ae8d1c-050b-4cac-b454-b67762c484bc admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.483183 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ae8d1c-050b-4cac-b454-b67762c484bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:54.485089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ae8d1c-050b-4cac-b454-b67762c484bc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:54.613655 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1359/2700] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:54 2022] POST /identity/v3/users => generated 290 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:24:54.620918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e1de50-c6d1-41e5-af7c-4ee15ebbd1ad admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:54.631610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e1de50-c6d1-41e5-af7c-4ee15ebbd1ad admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:54.636209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cb33b2-bd17-4e9b-9da3-3c031f5f888b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:54.636878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cb33b2-bd17-4e9b-9da3-3c031f5f888b admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.637222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cb33b2-bd17-4e9b-9da3-3c031f5f888b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.637555 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cb33b2-bd17-4e9b-9da3-3c031f5f888b admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.638275 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cb33b2-bd17-4e9b-9da3-3c031f5f888b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:54.639929 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cb33b2-bd17-4e9b-9da3-3c031f5f888b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:54.651427 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1342/2701] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:54.654918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae8d1c-050b-4cac-b454-b67762c484bc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:54.666346 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ae8d1c-050b-4cac-b454-b67762c484bc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:54.671167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09e61e1-38b5-43ea-a23d-ff5b2875f8c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:54.671881 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09e61e1-38b5-43ea-a23d-ff5b2875f8c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.672168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09e61e1-38b5-43ea-a23d-ff5b2875f8c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.672439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09e61e1-38b5-43ea-a23d-ff5b2875f8c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.673084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09e61e1-38b5-43ea-a23d-ff5b2875f8c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:54.674688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09e61e1-38b5-43ea-a23d-ff5b2875f8c9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:54.680365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:54.685472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1360/2702] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:24:54.690029 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cb33b2-bd17-4e9b-9da3-3c031f5f888b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:54.700140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cb33b2-bd17-4e9b-9da3-3c031f5f888b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:54.704698 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87ee99d-b4df-4d45-ae92-aefa2172376e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:54.705344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87ee99d-b4df-4d45-ae92-aefa2172376e admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.705644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87ee99d-b4df-4d45-ae92-aefa2172376e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.705918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87ee99d-b4df-4d45-ae92-aefa2172376e admin admin] PATH_INFO: `/v3/projects/bff73b9f76db4c1693e125af1d5044d8/users/474fd8f4871a43339f1b532ead96a899/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.721056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87ee99d-b4df-4d45-ae92-aefa2172376e admin admin] RBAC: Authorizing `identity:create_grant(project_id=bff73b9f76db4c1693e125af1d5044d8, user_id=474fd8f4871a43339f1b532ead96a899, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:54.724036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87ee99d-b4df-4d45-ae92-aefa2172376e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:54.819201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1343/2703] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:54 2022] PUT /identity/v3/projects/bff73b9f76db4c1693e125af1d5044d8/users/474fd8f4871a43339f1b532ead96a899/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:54.830462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f0fdb1-549d-4118-96dc-835750fa5f8e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.830751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f0fdb1-549d-4118-96dc-835750fa5f8e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.831054 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f0fdb1-549d-4118-96dc-835750fa5f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.859856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-83f0fdb1-549d-4118-96dc-835750fa5f8e None None] MFA Rules not processed for user `474fd8f4871a43339f1b532ead96a899`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:54.861037 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-83f0fdb1-549d-4118-96dc-835750fa5f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:54.924161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1361/2704] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:54.928632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87ee99d-b4df-4d45-ae92-aefa2172376e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:54.992051 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87ee99d-b4df-4d45-ae92-aefa2172376e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:54.996720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23309c7a-9e93-4fc8-ba04-96419e8fb871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:54.997319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23309c7a-9e93-4fc8-ba04-96419e8fb871 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:54.997625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23309c7a-9e93-4fc8-ba04-96419e8fb871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:54.997904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23309c7a-9e93-4fc8-ba04-96419e8fb871 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:54.998517 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23309c7a-9e93-4fc8-ba04-96419e8fb871 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:55.000408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23309c7a-9e93-4fc8-ba04-96419e8fb871 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:55.222525 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1344/2705] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:24:54 2022] POST /identity/v3/projects => generated 379 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:24:55.224914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f0fdb1-549d-4118-96dc-835750fa5f8e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:55.236460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f0fdb1-549d-4118-96dc-835750fa5f8e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:55.293309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9e96d1-dc00-4564-a739-fbc39c5ed2cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:55.293945 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9e96d1-dc00-4564-a739-fbc39c5ed2cc admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:55.294225 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9e96d1-dc00-4564-a739-fbc39c5ed2cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:55.294502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9e96d1-dc00-4564-a739-fbc39c5ed2cc admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:55.295125 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9e96d1-dc00-4564-a739-fbc39c5ed2cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:55.297017 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9e96d1-dc00-4564-a739-fbc39c5ed2cc admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:55.400582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1362/2706] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:24:55 2022] POST /identity/v3/users => generated 296 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:24:55.404668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23309c7a-9e93-4fc8-ba04-96419e8fb871 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:55.415004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23309c7a-9e93-4fc8-ba04-96419e8fb871 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:55.419808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24285769-9e34-444e-a20f-874a4fef676f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:55.420446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24285769-9e34-444e-a20f-874a4fef676f admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:55.420729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24285769-9e34-444e-a20f-874a4fef676f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:55.421001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24285769-9e34-444e-a20f-874a4fef676f admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:55.421680 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24285769-9e34-444e-a20f-874a4fef676f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:55.423366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24285769-9e34-444e-a20f-874a4fef676f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:55.434609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1345/2707] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:55.439012 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9e96d1-dc00-4564-a739-fbc39c5ed2cc admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:55.449363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9e96d1-dc00-4564-a739-fbc39c5ed2cc admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:55.453894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e93e39-0f1a-4051-b30c-a67597ae299c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:55.455448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e93e39-0f1a-4051-b30c-a67597ae299c admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:55.455448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e93e39-0f1a-4051-b30c-a67597ae299c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:55.455448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e93e39-0f1a-4051-b30c-a67597ae299c admin admin] PATH_INFO: `/v3/projects/ab141a4fdc2949df93935f7cf9cb75c6/users/b9c2c39d313a42728e6236499108758e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:55.469580 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e93e39-0f1a-4051-b30c-a67597ae299c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab141a4fdc2949df93935f7cf9cb75c6, user_id=b9c2c39d313a42728e6236499108758e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:55.472396 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e93e39-0f1a-4051-b30c-a67597ae299c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:55.550134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1363/2708] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:55 2022] PUT /identity/v3/projects/ab141a4fdc2949df93935f7cf9cb75c6/users/b9c2c39d313a42728e6236499108758e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:55.554156 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24285769-9e34-444e-a20f-874a4fef676f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:55.564832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24285769-9e34-444e-a20f-874a4fef676f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:55.621365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5aab7-19c3-4d75-bc52-ba0e99baf63d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:55.621999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5aab7-19c3-4d75-bc52-ba0e99baf63d admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:55.622283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5aab7-19c3-4d75-bc52-ba0e99baf63d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:55.622557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5aab7-19c3-4d75-bc52-ba0e99baf63d admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:55.623197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a5aab7-19c3-4d75-bc52-ba0e99baf63d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:55.624819 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a5aab7-19c3-4d75-bc52-ba0e99baf63d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:55.635255 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1346/2709] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:24:55 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:24:55.639403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e93e39-0f1a-4051-b30c-a67597ae299c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:55.649579 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e93e39-0f1a-4051-b30c-a67597ae299c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:55.654306 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:55.654891 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:55.655165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:55.655437 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] PATH_INFO: `/v3/projects/ab141a4fdc2949df93935f7cf9cb75c6/users/b9c2c39d313a42728e6236499108758e/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:55.657325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab141a4fdc2949df93935f7cf9cb75c6, user_id=b9c2c39d313a42728e6236499108758e, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:55.659972 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:55.670442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1364/2710] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:24:55 2022] PUT /identity/v3/projects/ab141a4fdc2949df93935f7cf9cb75c6/users/b9c2c39d313a42728e6236499108758e/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:24:55.680991 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffc105c-4e02-4176-b501-62268b426552 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:55.681269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffc105c-4e02-4176-b501-62268b426552 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:55.681541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffc105c-4e02-4176-b501-62268b426552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:55.708779 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8ffc105c-4e02-4176-b501-62268b426552 None None] MFA Rules not processed for user `b9c2c39d313a42728e6236499108758e`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:24:55.710015 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8ffc105c-4e02-4176-b501-62268b426552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:55.714918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:55.783102 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1347/2711] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:24:55 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:24:55.795391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:55.797108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:24:55.846819 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:24:55.859506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810188d3-a4b1-4c27-8233-40c8db202024 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:55.863101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a34f5a-878b-4a55-8f8c-b1d1d62b4c3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:55.863698 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a34f5a-878b-4a55-8f8c-b1d1d62b4c3b service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:55.863976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a34f5a-878b-4a55-8f8c-b1d1d62b4c3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:55.864248 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a34f5a-878b-4a55-8f8c-b1d1d62b4c3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:55.873770 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a34f5a-878b-4a55-8f8c-b1d1d62b4c3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:55.879537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a34f5a-878b-4a55-8f8c-b1d1d62b4c3b service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:55.885373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1365/2712] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:55 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:24:56.005616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffc105c-4e02-4176-b501-62268b426552 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:24:56.015145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffc105c-4e02-4176-b501-62268b426552 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:24:56.020477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d71e6d-12fd-457b-9e21-5f363921c9af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:24:56.020477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d71e6d-12fd-457b-9e21-5f363921c9af service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:24:56.020477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d71e6d-12fd-457b-9e21-5f363921c9af service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:24:56.020477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d71e6d-12fd-457b-9e21-5f363921c9af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:24:56.075842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d71e6d-12fd-457b-9e21-5f363921c9af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:24:56.077306 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d71e6d-12fd-457b-9e21-5f363921c9af service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:24:56.086607 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1348/2713] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:24:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:03.385624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a34f5a-878b-4a55-8f8c-b1d1d62b4c3b service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:03.447090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a34f5a-878b-4a55-8f8c-b1d1d62b4c3b service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:03.452161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:03.452779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:03.453053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:03.453321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] PATH_INFO: `/v3/users/6b411d8340e249ae806f49ce8c4056d6` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:03.454453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b411d8340e249ae806f49ce8c4056d6)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:03.456377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:03.648150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b411d8340e249ae806f49ce8c4056d6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:03.648827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b411d8340e249ae806f49ce8c4056d6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:03.661321 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b411d8340e249ae806f49ce8c4056d6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:03.775180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b411d8340e249ae806f49ce8c4056d6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:03.786097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b411d8340e249ae806f49ce8c4056d6'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:03.793387 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1366/2714] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:25:03 2022] DELETE /identity/v3/users/6b411d8340e249ae806f49ce8c4056d6 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:03.799610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925b6fa-9439-495d-9bb1-56221344a386 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:03.799610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925b6fa-9439-495d-9bb1-56221344a386 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:03.799610 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925b6fa-9439-495d-9bb1-56221344a386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:03.824370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2925b6fa-9439-495d-9bb1-56221344a386 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:03.825690 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2925b6fa-9439-495d-9bb1-56221344a386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:03.885958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1349/2715] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:03.897687 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:03.899283 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:03.955715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237b7ea-a01d-461f-8efa-fcaff2b260df admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:03.959653 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51891d70-8bbd-4bff-8f95-a877f2002cad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:03.960259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51891d70-8bbd-4bff-8f95-a877f2002cad service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:03.960535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51891d70-8bbd-4bff-8f95-a877f2002cad service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:03.960804 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51891d70-8bbd-4bff-8f95-a877f2002cad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:03.970421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51891d70-8bbd-4bff-8f95-a877f2002cad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:03.971872 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51891d70-8bbd-4bff-8f95-a877f2002cad service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:03.982133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1367/2716] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:25:04.419704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925b6fa-9439-495d-9bb1-56221344a386 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:04.421154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2925b6fa-9439-495d-9bb1-56221344a386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:04.433181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925b6fa-9439-495d-9bb1-56221344a386 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:04.438578 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:04.439296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:04.439573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:04.439839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] PATH_INFO: `/v3/projects/43de1928dee44836bcc5388a8a27cd9d` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:04.441715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] RBAC: Authorizing `identity:delete_project(project_id=43de1928dee44836bcc5388a8a27cd9d)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:04.443650 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:04.564168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:04.572464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43de1928dee44836bcc5388a8a27cd9d', 'request_id': 'req-069d86ca-d9b9-423b-8123-c7e93618bcee'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:04.664549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43de1928dee44836bcc5388a8a27cd9d', 'request_id': 'req-069d86ca-d9b9-423b-8123-c7e93618bcee'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:04.671407 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43de1928dee44836bcc5388a8a27cd9d', 'request_id': 'req-069d86ca-d9b9-423b-8123-c7e93618bcee'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:04.684472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] The token cache is being invalidate because project 43de1928dee44836bcc5388a8a27cd9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:25:04.684760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:04.685405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:04.686923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1350/2717] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:25:04 2022] DELETE /identity/v3/projects/43de1928dee44836bcc5388a8a27cd9d => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:04.691133 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51891d70-8bbd-4bff-8f95-a877f2002cad service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:04.701980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51891d70-8bbd-4bff-8f95-a877f2002cad service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:04.705499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:04.758758 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:04.759383 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:04.759665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:04.759948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] PATH_INFO: `/v3/users/474fd8f4871a43339f1b532ead96a899` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:04.761552 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=474fd8f4871a43339f1b532ead96a899)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:04.763483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:04.915922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '474fd8f4871a43339f1b532ead96a899'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:04.916613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '474fd8f4871a43339f1b532ead96a899'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:04.932813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '474fd8f4871a43339f1b532ead96a899'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:05.036069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '474fd8f4871a43339f1b532ead96a899'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:05.044162 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '474fd8f4871a43339f1b532ead96a899'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:05.050092 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1368/2718] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:25:04 2022] DELETE /identity/v3/users/474fd8f4871a43339f1b532ead96a899 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:25:05.516380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:05.517857 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:05.531296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069d86ca-d9b9-423b-8123-c7e93618bcee admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:05.591020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:05.591655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:05.591934 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:05.592206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] PATH_INFO: `/v3/projects/bff73b9f76db4c1693e125af1d5044d8` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:05.593322 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bff73b9f76db4c1693e125af1d5044d8)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:05.595268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:05.700881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bff73b9f76db4c1693e125af1d5044d8', 'request_id': 'req-b6a2e514-3397-41fe-a090-74f7fa325dc8'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:05.807529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bff73b9f76db4c1693e125af1d5044d8', 'request_id': 'req-b6a2e514-3397-41fe-a090-74f7fa325dc8'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:05.814611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bff73b9f76db4c1693e125af1d5044d8', 'request_id': 'req-b6a2e514-3397-41fe-a090-74f7fa325dc8'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:05.826669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] The token cache is being invalidate because project bff73b9f76db4c1693e125af1d5044d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:25:05.826952 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:05.827586 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:05.829099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1351/2719] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:25:05 2022] DELETE /identity/v3/projects/bff73b9f76db4c1693e125af1d5044d8 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:05.833160 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:05.834669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:05.847313 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fee9db-8e30-4fce-b847-196788390a0c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:05.906563 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:05.907230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:05.907512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:05.907786 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] PATH_INFO: `/v3/users/b9c2c39d313a42728e6236499108758e` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:05.908953 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9c2c39d313a42728e6236499108758e)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:05.910897 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:06.026416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:06.140184 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9c2c39d313a42728e6236499108758e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:06.140856 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9c2c39d313a42728e6236499108758e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:06.152741 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9c2c39d313a42728e6236499108758e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:06.218497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9c2c39d313a42728e6236499108758e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:06.228497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0b80ea02-64f8-48ba-b91d-1da3a37e332d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9c2c39d313a42728e6236499108758e'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:06.237316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1369/2720] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:25:05 2022] DELETE /identity/v3/users/b9c2c39d313a42728e6236499108758e => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:25:06.711704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:06.713790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:06.728123 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a2e514-3397-41fe-a090-74f7fa325dc8 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:06.788424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:06.796653 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:06.797317 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:06.797635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:06.798234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] PATH_INFO: `/v3/projects/ab141a4fdc2949df93935f7cf9cb75c6` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:06.799162 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab141a4fdc2949df93935f7cf9cb75c6)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:06.801166 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:06.937651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab141a4fdc2949df93935f7cf9cb75c6', 'request_id': 'req-8e39644a-87f0-4be8-bf4c-16f164ba6695'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:07.045193 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab141a4fdc2949df93935f7cf9cb75c6', 'request_id': 'req-8e39644a-87f0-4be8-bf4c-16f164ba6695'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:07.050422 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab141a4fdc2949df93935f7cf9cb75c6', 'request_id': 'req-8e39644a-87f0-4be8-bf4c-16f164ba6695'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:07.061846 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] The token cache is being invalidate because project ab141a4fdc2949df93935f7cf9cb75c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:25:07.062290 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:07.062948 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8e39644a-87f0-4be8-bf4c-16f164ba6695 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:07.064485 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1352/2721] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:25:06 2022] DELETE /identity/v3/projects/ab141a4fdc2949df93935f7cf9cb75c6 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:07.070731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfaa733-6a7d-48cb-919b-9156e4df8ef3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.071187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfaa733-6a7d-48cb-919b-9156e4df8ef3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.071300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfaa733-6a7d-48cb-919b-9156e4df8ef3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.092284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-edfaa733-6a7d-48cb-919b-9156e4df8ef3 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:07.093479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-edfaa733-6a7d-48cb-919b-9156e4df8ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:07.153732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1370/2722] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:07.162247 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226bcb00-b4dd-4cad-8054-c06c9c622f84 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.162536 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226bcb00-b4dd-4cad-8054-c06c9c622f84 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.162811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226bcb00-b4dd-4cad-8054-c06c9c622f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.182091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-226bcb00-b4dd-4cad-8054-c06c9c622f84 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:07.183333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-226bcb00-b4dd-4cad-8054-c06c9c622f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:07.191239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1353/2723] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:07.195460 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfaa733-6a7d-48cb-919b-9156e4df8ef3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:07.205541 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfaa733-6a7d-48cb-919b-9156e4df8ef3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:07.210453 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:07.211057 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.211336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.211609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.212370 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:25:07.212773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:07.215157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:07.227550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1371/2724] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:25:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:25:07.232083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226bcb00-b4dd-4cad-8054-c06c9c622f84 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:07.241344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226bcb00-b4dd-4cad-8054-c06c9c622f84 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:07.246775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f4dbb0-1442-4c38-a859-52ab932ae941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:07.247345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f4dbb0-1442-4c38-a859-52ab932ae941 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.247619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f4dbb0-1442-4c38-a859-52ab932ae941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.247884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f4dbb0-1442-4c38-a859-52ab932ae941 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.248543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f4dbb0-1442-4c38-a859-52ab932ae941 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:07.250539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f4dbb0-1442-4c38-a859-52ab932ae941 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:07.363619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1354/2725] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:25:07 2022] POST /identity/v3/projects => generated 385 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:25:07.367471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:07.377918 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5656ee5-737f-4218-a950-ec003a5096ce admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:07.435796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d3b4e4-0a55-452b-b348-7c4c94705c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:07.436402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d3b4e4-0a55-452b-b348-7c4c94705c4f admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.436678 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d3b4e4-0a55-452b-b348-7c4c94705c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.436946 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d3b4e4-0a55-452b-b348-7c4c94705c4f admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.437568 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d3b4e4-0a55-452b-b348-7c4c94705c4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:07.439501 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d3b4e4-0a55-452b-b348-7c4c94705c4f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:07.568801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1372/2726] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:25:07 2022] POST /identity/v3/users => generated 293 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:25:07.572824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f4dbb0-1442-4c38-a859-52ab932ae941 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:07.584091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f4dbb0-1442-4c38-a859-52ab932ae941 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:07.588781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2696667-047b-48b0-8848-b65087bfeb0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:07.589431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2696667-047b-48b0-8848-b65087bfeb0d admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.589752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2696667-047b-48b0-8848-b65087bfeb0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.590024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2696667-047b-48b0-8848-b65087bfeb0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.590662 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2696667-047b-48b0-8848-b65087bfeb0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:07.592390 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2696667-047b-48b0-8848-b65087bfeb0d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:07.602827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1355/2727] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:07.606919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d3b4e4-0a55-452b-b348-7c4c94705c4f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:07.617027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d3b4e4-0a55-452b-b348-7c4c94705c4f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:07.622230 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2e8384-b893-4621-aa9b-cd8dc0bf2bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:07.622871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2e8384-b893-4621-aa9b-cd8dc0bf2bce admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.623147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2e8384-b893-4621-aa9b-cd8dc0bf2bce admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.623421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2e8384-b893-4621-aa9b-cd8dc0bf2bce admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.624060 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2e8384-b893-4621-aa9b-cd8dc0bf2bce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:07.625717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2e8384-b893-4621-aa9b-cd8dc0bf2bce admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:07.636471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1373/2728] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:07 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:07.640499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2696667-047b-48b0-8848-b65087bfeb0d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:07.650417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2696667-047b-48b0-8848-b65087bfeb0d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:07.654988 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a43f0d8-9b7d-4e38-a2e9-93607bbe1298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:07.655732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43f0d8-9b7d-4e38-a2e9-93607bbe1298 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.656011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43f0d8-9b7d-4e38-a2e9-93607bbe1298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.656281 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43f0d8-9b7d-4e38-a2e9-93607bbe1298 admin admin] PATH_INFO: `/v3/projects/81b653b174ed4838b25d8a1a59d8bf50/users/ad7be9522fd84a6bab11601e3ee889c7/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.671195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a43f0d8-9b7d-4e38-a2e9-93607bbe1298 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81b653b174ed4838b25d8a1a59d8bf50, user_id=ad7be9522fd84a6bab11601e3ee889c7, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:07.673964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a43f0d8-9b7d-4e38-a2e9-93607bbe1298 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:07.778110 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1356/2729] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:25:07 2022] PUT /identity/v3/projects/81b653b174ed4838b25d8a1a59d8bf50/users/ad7be9522fd84a6bab11601e3ee889c7/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:07.790332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77379ba-a9ea-4916-89ec-71bba4123f39 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.790660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77379ba-a9ea-4916-89ec-71bba4123f39 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.790938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77379ba-a9ea-4916-89ec-71bba4123f39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.812737 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:07.819767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-b77379ba-a9ea-4916-89ec-71bba4123f39 None None] MFA Rules not processed for user `ad7be9522fd84a6bab11601e3ee889c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:07.823836 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b77379ba-a9ea-4916-89ec-71bba4123f39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:07.884164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1374/2730] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:07.888800 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a43f0d8-9b7d-4e38-a2e9-93607bbe1298 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:07.953097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a43f0d8-9b7d-4e38-a2e9-93607bbe1298 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:07.957951 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aba7b8-916b-4459-86d7-9ecf18afcffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:07.958670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aba7b8-916b-4459-86d7-9ecf18afcffb admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:07.958957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aba7b8-916b-4459-86d7-9ecf18afcffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:07.959234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aba7b8-916b-4459-86d7-9ecf18afcffb admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:07.959867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33aba7b8-916b-4459-86d7-9ecf18afcffb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:07.961864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33aba7b8-916b-4459-86d7-9ecf18afcffb admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:08.112724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1357/2731] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:25:07 2022] POST /identity/v3/projects => generated 385 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:25:08.116760 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77379ba-a9ea-4916-89ec-71bba4123f39 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:08.127629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77379ba-a9ea-4916-89ec-71bba4123f39 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:08.184206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41cc33e-2c85-4d54-9650-14dfa45322f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:08.184750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41cc33e-2c85-4d54-9650-14dfa45322f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:08.185029 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41cc33e-2c85-4d54-9650-14dfa45322f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:08.185298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41cc33e-2c85-4d54-9650-14dfa45322f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:08.185951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41cc33e-2c85-4d54-9650-14dfa45322f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:08.187848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41cc33e-2c85-4d54-9650-14dfa45322f5 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:08.299377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1375/2732] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:25:08 2022] POST /identity/v3/users => generated 299 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:25:08.303371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aba7b8-916b-4459-86d7-9ecf18afcffb admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:08.313806 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aba7b8-916b-4459-86d7-9ecf18afcffb admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:08.318379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a15f67-4e55-43b5-95d5-294b8680fa23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:08.318975 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a15f67-4e55-43b5-95d5-294b8680fa23 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:08.319252 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a15f67-4e55-43b5-95d5-294b8680fa23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:08.319544 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a15f67-4e55-43b5-95d5-294b8680fa23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:08.320205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a15f67-4e55-43b5-95d5-294b8680fa23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:08.321842 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a15f67-4e55-43b5-95d5-294b8680fa23 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:08.323117 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:08.332655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1358/2733] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:08 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:08.336719 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41cc33e-2c85-4d54-9650-14dfa45322f5 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:08.347326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41cc33e-2c85-4d54-9650-14dfa45322f5 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:08.352419 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b85a16-ef98-46ec-839e-c9d36090a964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:08.353064 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b85a16-ef98-46ec-839e-c9d36090a964 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:08.353343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b85a16-ef98-46ec-839e-c9d36090a964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:08.353639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b85a16-ef98-46ec-839e-c9d36090a964 admin admin] PATH_INFO: `/v3/projects/91fe28491b1246ed818b24c9ead518fa/users/e5ba36a7d8de40cdb5ce19878a8375c7/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:08.368965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b85a16-ef98-46ec-839e-c9d36090a964 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91fe28491b1246ed818b24c9ead518fa, user_id=e5ba36a7d8de40cdb5ce19878a8375c7, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:08.371624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b85a16-ef98-46ec-839e-c9d36090a964 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:08.493111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1376/2734] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:25:08 2022] PUT /identity/v3/projects/91fe28491b1246ed818b24c9ead518fa/users/e5ba36a7d8de40cdb5ce19878a8375c7/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:25:08.497027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a15f67-4e55-43b5-95d5-294b8680fa23 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:08.506791 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a15f67-4e55-43b5-95d5-294b8680fa23 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:08.562835 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8887c6-d037-4356-a876-e7e2deabd4c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:08.563439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8887c6-d037-4356-a876-e7e2deabd4c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:08.563720 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8887c6-d037-4356-a876-e7e2deabd4c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:08.563994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8887c6-d037-4356-a876-e7e2deabd4c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:08.564622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8887c6-d037-4356-a876-e7e2deabd4c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:08.566305 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8887c6-d037-4356-a876-e7e2deabd4c2 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:08.576445 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1359/2735] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:08 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:08.580487 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b85a16-ef98-46ec-839e-c9d36090a964 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:08.591663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b85a16-ef98-46ec-839e-c9d36090a964 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:08.596268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:08.596913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:08.597189 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:08.597462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] PATH_INFO: `/v3/projects/91fe28491b1246ed818b24c9ead518fa/users/e5ba36a7d8de40cdb5ce19878a8375c7/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:08.599637 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91fe28491b1246ed818b24c9ead518fa, user_id=e5ba36a7d8de40cdb5ce19878a8375c7, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:08.602289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:08.612286 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1377/2736] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:25:08 2022] PUT /identity/v3/projects/91fe28491b1246ed818b24c9ead518fa/users/e5ba36a7d8de40cdb5ce19878a8375c7/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:08.622367 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99a1dd4-4061-46e5-99eb-10824c7059da None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:08.622685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99a1dd4-4061-46e5-99eb-10824c7059da None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:08.623049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99a1dd4-4061-46e5-99eb-10824c7059da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:08.650499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-d99a1dd4-4061-46e5-99eb-10824c7059da None None] MFA Rules not processed for user `e5ba36a7d8de40cdb5ce19878a8375c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:08.652346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d99a1dd4-4061-46e5-99eb-10824c7059da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:08.722170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1360/2737] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:08 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:08.733699 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:08.735391 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:08.798404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a53ae7-0e14-43bc-8388-5314231549b0 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:08.802027 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:08.802660 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:08.802938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:08.803209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:08.808844 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:08.860615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:08.862108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:08.872165 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1378/2738] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:09.718251 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99a1dd4-4061-46e5-99eb-10824c7059da None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:09.720024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d99a1dd4-4061-46e5-99eb-10824c7059da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:09.806908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99a1dd4-4061-46e5-99eb-10824c7059da None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:09.811989 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:09.812629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:09.812911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:09.813184 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:09.814957 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:09.827530 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:09.829115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:09.840912 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1361/2739] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:11.125895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:11.127378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:11.195649 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2edd0d3-0f9b-4504-ad26-5d38580c6962 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:11.200068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:11.200794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:11.201099 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:11.201373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] PATH_INFO: `/v3/users/3d598ffecf5744ccb038e088f039f037` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:11.202536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d598ffecf5744ccb038e088f039f037)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:11.204448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:11.442979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d598ffecf5744ccb038e088f039f037'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:11.443149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d598ffecf5744ccb038e088f039f037'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:11.457011 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d598ffecf5744ccb038e088f039f037'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:11.527327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d598ffecf5744ccb038e088f039f037'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:11.535420 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d598ffecf5744ccb038e088f039f037'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:11.541310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1379/2740] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:25:11 2022] DELETE /identity/v3/users/3d598ffecf5744ccb038e088f039f037 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:11.778611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:11.783853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=100 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:11.790406 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2217f9a6-1b04-418f-a580-4fc8018d4bde service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:11.850083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:11.850741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:11.851026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:11.851299 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] PATH_INFO: `/v3/projects/684e3adc0a9148cda29de351c4f066fe` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:11.852602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=684e3adc0a9148cda29de351c4f066fe)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:11.854480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:11.980944 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '684e3adc0a9148cda29de351c4f066fe', 'request_id': 'req-5631671d-52a4-4087-835a-d217a4f29b9e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.091848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '684e3adc0a9148cda29de351c4f066fe', 'request_id': 'req-5631671d-52a4-4087-835a-d217a4f29b9e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.098654 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '684e3adc0a9148cda29de351c4f066fe', 'request_id': 'req-5631671d-52a4-4087-835a-d217a4f29b9e'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.110150 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] The token cache is being invalidate because project 684e3adc0a9148cda29de351c4f066fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:25:12.110441 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.111076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.112592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1362/2741] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:25:11 2022] DELETE /identity/v3/projects/684e3adc0a9148cda29de351c4f066fe => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:12.116718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:12.118126 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:12.131388 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e00e1d5-2030-4090-94e4-a6453b9eb7e4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:12.155732 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:12.187690 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:12.188327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:12.188605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:12.188920 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] PATH_INFO: `/v3/users/14354f56ed6f4ec8a9273c9c80d2faee` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:12.190142 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14354f56ed6f4ec8a9273c9c80d2faee)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:12.192167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:12.512011 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14354f56ed6f4ec8a9273c9c80d2faee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.512647 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14354f56ed6f4ec8a9273c9c80d2faee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.524052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14354f56ed6f4ec8a9273c9c80d2faee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.624252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14354f56ed6f4ec8a9273c9c80d2faee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.631714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d6c4f58a-4431-4422-9eac-9bdec7602538 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14354f56ed6f4ec8a9273c9c80d2faee'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:12.638223 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1380/2742] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:25:12 2022] DELETE /identity/v3/users/14354f56ed6f4ec8a9273c9c80d2faee => generated 0 bytes in 522 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:12.900114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:12.901673 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:12.914811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5631671d-52a4-4087-835a-d217a4f29b9e admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:12.972138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:12.972891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:12.973175 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:12.973752 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] PATH_INFO: `/v3/projects/064e370f28e8496a9ab3f6e93a3ef03f` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:12.974880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=064e370f28e8496a9ab3f6e93a3ef03f)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:12.976794 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:13.093866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:13.102976 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '064e370f28e8496a9ab3f6e93a3ef03f', 'request_id': 'req-8fd8357a-3266-4b53-9d5f-4a8d8838af20'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:13.178596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '064e370f28e8496a9ab3f6e93a3ef03f', 'request_id': 'req-8fd8357a-3266-4b53-9d5f-4a8d8838af20'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:13.184797 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '064e370f28e8496a9ab3f6e93a3ef03f', 'request_id': 'req-8fd8357a-3266-4b53-9d5f-4a8d8838af20'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:13.196124 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] The token cache is being invalidate because project 064e370f28e8496a9ab3f6e93a3ef03f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:25:13.196571 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:13.197049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-8fd8357a-3266-4b53-9d5f-4a8d8838af20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:13.198618 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1363/2743] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:25:12 2022] DELETE /identity/v3/projects/064e370f28e8496a9ab3f6e93a3ef03f => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:13.203512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a4308-a2c6-445a-b943-c5e5b247f113 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.204052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a4308-a2c6-445a-b943-c5e5b247f113 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.204158 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a4308-a2c6-445a-b943-c5e5b247f113 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.224934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-249a4308-a2c6-445a-b943-c5e5b247f113 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:13.226169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-249a4308-a2c6-445a-b943-c5e5b247f113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:13.284882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1381/2744] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:13.294299 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac3cfdd-d654-4331-9b20-aaa80844e932 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.294588 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac3cfdd-d654-4331-9b20-aaa80844e932 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.294864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac3cfdd-d654-4331-9b20-aaa80844e932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.314235 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5ac3cfdd-d654-4331-9b20-aaa80844e932 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:13.315430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5ac3cfdd-d654-4331-9b20-aaa80844e932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:13.323310 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1364/2745] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:25:13.327402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a4308-a2c6-445a-b943-c5e5b247f113 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:13.336919 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a4308-a2c6-445a-b943-c5e5b247f113 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:13.342033 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:13.342624 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.342900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.343169 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.343842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:25:13.344140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:13.345743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:13.357868 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1382/2746] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:25:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:25:13.361958 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac3cfdd-d654-4331-9b20-aaa80844e932 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:13.370977 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac3cfdd-d654-4331-9b20-aaa80844e932 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:13.375627 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5823cd-2a32-4b40-94d3-bd8df7cc20f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:13.376239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5823cd-2a32-4b40-94d3-bd8df7cc20f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.376522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5823cd-2a32-4b40-94d3-bd8df7cc20f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.376798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5823cd-2a32-4b40-94d3-bd8df7cc20f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.377424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5823cd-2a32-4b40-94d3-bd8df7cc20f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:13.379357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5823cd-2a32-4b40-94d3-bd8df7cc20f3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:13.476966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1365/2747] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:25:13 2022] POST /identity/v3/projects => generated 377 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:25:13.481583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:13.491577 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f2970-32fe-477b-9cec-67aa1a42602f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:13.520876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:13.547724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae4d9c-7c62-4697-b4ad-324a26c0669c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:13.548325 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae4d9c-7c62-4697-b4ad-324a26c0669c admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.548602 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae4d9c-7c62-4697-b4ad-324a26c0669c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.548871 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae4d9c-7c62-4697-b4ad-324a26c0669c admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.549497 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ae4d9c-7c62-4697-b4ad-324a26c0669c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:13.551426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ae4d9c-7c62-4697-b4ad-324a26c0669c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:13.700203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1383/2748] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:25:13 2022] POST /identity/v3/users => generated 289 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:25:13.704327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5823cd-2a32-4b40-94d3-bd8df7cc20f3 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:13.715115 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5823cd-2a32-4b40-94d3-bd8df7cc20f3 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:13.719661 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08af60f3-b9b1-4232-aa80-ee32dcc85964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:13.720269 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af60f3-b9b1-4232-aa80-ee32dcc85964 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.720545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af60f3-b9b1-4232-aa80-ee32dcc85964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.720818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08af60f3-b9b1-4232-aa80-ee32dcc85964 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.721453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08af60f3-b9b1-4232-aa80-ee32dcc85964 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:13.723076 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08af60f3-b9b1-4232-aa80-ee32dcc85964 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:13.733234 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1366/2749] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:13.737082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae4d9c-7c62-4697-b4ad-324a26c0669c admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:13.746976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae4d9c-7c62-4697-b4ad-324a26c0669c admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:13.751987 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232bab48-4c9b-4333-9d69-bab046b40ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:13.752572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232bab48-4c9b-4333-9d69-bab046b40ffd admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.752848 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232bab48-4c9b-4333-9d69-bab046b40ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.753118 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232bab48-4c9b-4333-9d69-bab046b40ffd admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.753793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232bab48-4c9b-4333-9d69-bab046b40ffd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:13.755360 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232bab48-4c9b-4333-9d69-bab046b40ffd admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:13.765578 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1384/2750] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:13 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:25:13.769385 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08af60f3-b9b1-4232-aa80-ee32dcc85964 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:13.780023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08af60f3-b9b1-4232-aa80-ee32dcc85964 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:13.784495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:13.785135 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.785417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.785729 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] PATH_INFO: `/v3/projects/bc9eef4ecfd04739beac5344bcec4271/users/d9027c2734634603a7c8b4990b790300/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.800266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc9eef4ecfd04739beac5344bcec4271, user_id=d9027c2734634603a7c8b4990b790300, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:13.802954 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:13.870178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1367/2751] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:25:13 2022] PUT /identity/v3/projects/bc9eef4ecfd04739beac5344bcec4271/users/d9027c2734634603a7c8b4990b790300/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:13.880377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe67c6-7d6d-4151-91cf-57ceee657132 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:13.880665 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe67c6-7d6d-4151-91cf-57ceee657132 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:13.880966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbe67c6-7d6d-4151-91cf-57ceee657132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:13.909063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-6bbe67c6-7d6d-4151-91cf-57ceee657132 None None] MFA Rules not processed for user `d9027c2734634603a7c8b4990b790300`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:13.910289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6bbe67c6-7d6d-4151-91cf-57ceee657132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:13.972730 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1385/2752] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:25:13.984139 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:13.985867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:14.064245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed02dad9-9023-4a16-90bd-d2bbd3d65545 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:14.067891 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3d4d5a-4e6e-4796-b5d2-48bdda826a09 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6cc8f34f9674a94ba23a7157e44a6f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:14.068507 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d4d5a-4e6e-4796-b5d2-48bdda826a09 service glance] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:14.068784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d4d5a-4e6e-4796-b5d2-48bdda826a09 service glance] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:14.069052 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3d4d5a-4e6e-4796-b5d2-48bdda826a09 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:14.077764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3d4d5a-4e6e-4796-b5d2-48bdda826a09 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:14.079446 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3d4d5a-4e6e-4796-b5d2-48bdda826a09 service glance] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:14.090229 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1368/2753] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:13 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:19.717210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbe67c6-7d6d-4151-91cf-57ceee657132 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:19.718728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6bbe67c6-7d6d-4151-91cf-57ceee657132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:19.775870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbe67c6-7d6d-4151-91cf-57ceee657132 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:19.780203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:19.780807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:19.781089 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:19.781362 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:19.783226 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:19.834310 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:19.846675 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:19.848113 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:19.861022 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1386/2754] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:19 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:24.084195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=5,keystone:SELECT=73,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:26.070472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d82278-efe9-40be-9a30-ecc8c2a91f29 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:26.071144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d82278-efe9-40be-9a30-ecc8c2a91f29 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:26.071144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d82278-efe9-40be-9a30-ecc8c2a91f29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:26.090950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-30d82278-efe9-40be-9a30-ecc8c2a91f29 None None] MFA Rules not processed for user `6a4136b08cc1464eac6c779652a1d2b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:26.092201 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-30d82278-efe9-40be-9a30-ecc8c2a91f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:26.150635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1369/2755] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:26.392999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:26.394511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:26.452820 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b5412-8849-4d17-973e-0d6baec7ebeb service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:26.456429 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:26.457024 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:26.457298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:26.457566 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:26.459511 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:26.470626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:26.472076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:26.481151 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1387/2756] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:26.915176 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d82278-efe9-40be-9a30-ecc8c2a91f29 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:26.916692 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-30d82278-efe9-40be-9a30-ecc8c2a91f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:27.001505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d82278-efe9-40be-9a30-ecc8c2a91f29 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:27.006694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:27.007311 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:27.007593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:27.007868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:27.009866 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:27.022437 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:27.024094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:27.034564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1370/2757] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:30.234204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:30.235608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:30.246021 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e753399c-5981-4172-bc79-9e4ee4f42766 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:30.250762 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d1a181-02ea-4a91-9960-82dab069c199 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:30.251794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d1a181-02ea-4a91-9960-82dab069c199 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:30.252302 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d1a181-02ea-4a91-9960-82dab069c199 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:30.255682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d1a181-02ea-4a91-9960-82dab069c199 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:30.255682 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-b8d1a181-02ea-4a91-9960-82dab069c199 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:30.266053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d1a181-02ea-4a91-9960-82dab069c199 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:30.267483 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d1a181-02ea-4a91-9960-82dab069c199 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:30.275767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1388/2758] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:30 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:31.073373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:31.073981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:31.149370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66605ac3-8f3a-417d-9e65-a44730896d7d service placement] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:31.154094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:31.154724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:31.155007 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:31.155282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] PATH_INFO: `/v3/users/ad7be9522fd84a6bab11601e3ee889c7` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:31.157028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad7be9522fd84a6bab11601e3ee889c7)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:31.158997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:31.321403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:31.330810 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad7be9522fd84a6bab11601e3ee889c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:31.346366 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad7be9522fd84a6bab11601e3ee889c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:31.347125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad7be9522fd84a6bab11601e3ee889c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:31.458253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad7be9522fd84a6bab11601e3ee889c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:31.469249 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad7be9522fd84a6bab11601e3ee889c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:31.475943 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1371/2759] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:25:31 2022] DELETE /identity/v3/users/ad7be9522fd84a6bab11601e3ee889c7 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:31.480498 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1757374-38dd-4ad6-84d1-1bb17ec5694f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:31.481307 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1757374-38dd-4ad6-84d1-1bb17ec5694f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:31.481583 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1757374-38dd-4ad6-84d1-1bb17ec5694f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:31.502191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-f1757374-38dd-4ad6-84d1-1bb17ec5694f None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:31.503380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f1757374-38dd-4ad6-84d1-1bb17ec5694f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:31.593175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1389/2760] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:31.603168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:31.604642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:31.663476 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9768151-5425-45df-a23b-55a53454ae2b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:31.667268 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8438e6-0eb6-40c4-99ab-2b3a3459afd3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:31.667872 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8438e6-0eb6-40c4-99ab-2b3a3459afd3 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:31.668148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8438e6-0eb6-40c4-99ab-2b3a3459afd3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:31.668417 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8438e6-0eb6-40c4-99ab-2b3a3459afd3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:31.677561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8438e6-0eb6-40c4-99ab-2b3a3459afd3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:31.679024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8438e6-0eb6-40c4-99ab-2b3a3459afd3 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:31.689178 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1372/2761] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:32.140222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1757374-38dd-4ad6-84d1-1bb17ec5694f None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:32.141773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f1757374-38dd-4ad6-84d1-1bb17ec5694f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:32.153727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1757374-38dd-4ad6-84d1-1bb17ec5694f None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:32.158589 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:32.159234 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:32.160039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:32.160039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] PATH_INFO: `/v3/projects/81b653b174ed4838b25d8a1a59d8bf50` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:32.161140 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] RBAC: Authorizing `identity:delete_project(project_id=81b653b174ed4838b25d8a1a59d8bf50)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:32.163120 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:32.287366 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81b653b174ed4838b25d8a1a59d8bf50', 'request_id': 'req-cd0a7456-8b2c-44ef-afd5-71a20089449f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.292816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81b653b174ed4838b25d8a1a59d8bf50', 'request_id': 'req-cd0a7456-8b2c-44ef-afd5-71a20089449f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.416220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81b653b174ed4838b25d8a1a59d8bf50', 'request_id': 'req-cd0a7456-8b2c-44ef-afd5-71a20089449f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.428249 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] The token cache is being invalidate because project 81b653b174ed4838b25d8a1a59d8bf50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:25:32.428537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.429179 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.435078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1390/2762] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:25:32 2022] DELETE /identity/v3/projects/81b653b174ed4838b25d8a1a59d8bf50 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:32.435209 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8438e6-0eb6-40c4-99ab-2b3a3459afd3 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:32.444490 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8438e6-0eb6-40c4-99ab-2b3a3459afd3 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:32.501923 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:32.502542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:32.502820 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:32.503092 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] PATH_INFO: `/v3/users/e5ba36a7d8de40cdb5ce19878a8375c7` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:32.504187 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5ba36a7d8de40cdb5ce19878a8375c7)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:32.506114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:32.718557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5ba36a7d8de40cdb5ce19878a8375c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.727164 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:32.731513 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5ba36a7d8de40cdb5ce19878a8375c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.732172 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5ba36a7d8de40cdb5ce19878a8375c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.800926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5ba36a7d8de40cdb5ce19878a8375c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.808760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d40b4596-a106-4d4f-a077-f6342929a450 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5ba36a7d8de40cdb5ce19878a8375c7'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:32.815357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1373/2763] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:25:32 2022] DELETE /identity/v3/users/e5ba36a7d8de40cdb5ce19878a8375c7 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:33.288216 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:33.289796 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:33.300715 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:33.303753 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0a7456-8b2c-44ef-afd5-71a20089449f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:33.361929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:33.362555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:33.362832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:33.363108 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] PATH_INFO: `/v3/projects/91fe28491b1246ed818b24c9ead518fa` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:33.364206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] RBAC: Authorizing `identity:delete_project(project_id=91fe28491b1246ed818b24c9ead518fa)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:33.366122 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:33.480839 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91fe28491b1246ed818b24c9ead518fa', 'request_id': 'req-05501f2d-6c53-42b6-a5b1-df8749934255'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:33.486639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91fe28491b1246ed818b24c9ead518fa', 'request_id': 'req-05501f2d-6c53-42b6-a5b1-df8749934255'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:33.588170 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91fe28491b1246ed818b24c9ead518fa', 'request_id': 'req-05501f2d-6c53-42b6-a5b1-df8749934255'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:33.600087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] The token cache is being invalidate because project 91fe28491b1246ed818b24c9ead518fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:25:33.602996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:33.602996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-05501f2d-6c53-42b6-a5b1-df8749934255 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:33.603893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1391/2764] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:25:33 2022] DELETE /identity/v3/projects/91fe28491b1246ed818b24c9ead518fa => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:25:33.610442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb6bd6c-f0cc-431f-b78b-15a2894779a2 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:33.610915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb6bd6c-f0cc-431f-b78b-15a2894779a2 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:33.611031 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb6bd6c-f0cc-431f-b78b-15a2894779a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:33.630775 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8fb6bd6c-f0cc-431f-b78b-15a2894779a2 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:33.634761 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8fb6bd6c-f0cc-431f-b78b-15a2894779a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:33.690173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1374/2765] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:33.756109 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7314fc-f964-452a-ab5c-251e600aec48 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:33.756392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7314fc-f964-452a-ab5c-251e600aec48 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:33.756657 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7314fc-f964-452a-ab5c-251e600aec48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:33.777507 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4d7314fc-f964-452a-ab5c-251e600aec48 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:33.778724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4d7314fc-f964-452a-ab5c-251e600aec48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:33.787638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1392/2766] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:33.791637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb6bd6c-f0cc-431f-b78b-15a2894779a2 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:33.801732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb6bd6c-f0cc-431f-b78b-15a2894779a2 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:33.806331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:33.806930 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:33.807210 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:33.807478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:33.808129 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:25:33.808436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:33.810055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:33.821486 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1375/2767] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:25:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:25:33.825971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7314fc-f964-452a-ab5c-251e600aec48 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:33.835342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7314fc-f964-452a-ab5c-251e600aec48 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:33.840026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb38dea-a41d-47da-b655-1cfcc88dfab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:33.840626 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb38dea-a41d-47da-b655-1cfcc88dfab4 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:33.840905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb38dea-a41d-47da-b655-1cfcc88dfab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:33.841177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb38dea-a41d-47da-b655-1cfcc88dfab4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:33.841841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb38dea-a41d-47da-b655-1cfcc88dfab4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:33.843737 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb38dea-a41d-47da-b655-1cfcc88dfab4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:33.939663 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1393/2768] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:25:33 2022] POST /identity/v3/projects => generated 375 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:25:33.944096 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:33.954212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47d5e19-a65c-41f3-bf90-003ec45b0c6d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:34.014764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6af1413-c7d5-4195-a2f5-5e22b67365dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:34.017378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6af1413-c7d5-4195-a2f5-5e22b67365dd admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:34.017378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6af1413-c7d5-4195-a2f5-5e22b67365dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:34.017378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6af1413-c7d5-4195-a2f5-5e22b67365dd admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:34.017378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6af1413-c7d5-4195-a2f5-5e22b67365dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:34.019182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6af1413-c7d5-4195-a2f5-5e22b67365dd admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:34.120788 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1376/2769] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:25:33 2022] POST /identity/v3/users => generated 288 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:25:34.124900 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb38dea-a41d-47da-b655-1cfcc88dfab4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:34.135718 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb38dea-a41d-47da-b655-1cfcc88dfab4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:34.140434 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61f7ca8-7b53-4bad-add5-1f62d7f1b1c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:34.143472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f7ca8-7b53-4bad-add5-1f62d7f1b1c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:34.143472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f7ca8-7b53-4bad-add5-1f62d7f1b1c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:34.143472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f7ca8-7b53-4bad-add5-1f62d7f1b1c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:34.143472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61f7ca8-7b53-4bad-add5-1f62d7f1b1c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:34.144726 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61f7ca8-7b53-4bad-add5-1f62d7f1b1c9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:34.154684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1394/2770] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:34.158696 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6af1413-c7d5-4195-a2f5-5e22b67365dd admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:34.168429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6af1413-c7d5-4195-a2f5-5e22b67365dd admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:34.173461 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b856cbc2-4e0e-4c8d-ba8a-da1905b69c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:34.174082 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b856cbc2-4e0e-4c8d-ba8a-da1905b69c0b admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:34.174357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b856cbc2-4e0e-4c8d-ba8a-da1905b69c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:34.174624 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b856cbc2-4e0e-4c8d-ba8a-da1905b69c0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:34.175248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b856cbc2-4e0e-4c8d-ba8a-da1905b69c0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:34.176833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b856cbc2-4e0e-4c8d-ba8a-da1905b69c0b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:34.186629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1377/2771] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:34 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:34.191114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61f7ca8-7b53-4bad-add5-1f62d7f1b1c9 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:34.200865 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61f7ca8-7b53-4bad-add5-1f62d7f1b1c9 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:34.205399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:34.206074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:34.206365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:34.206638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] PATH_INFO: `/v3/projects/68b2ebaa92044f5b96e0e6ca1d3c99a2/users/b563c1dc9f8943669879e62c10a23854/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:34.220916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=68b2ebaa92044f5b96e0e6ca1d3c99a2, user_id=b563c1dc9f8943669879e62c10a23854, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:34.223841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:34.304091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1395/2772] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:25:34 2022] PUT /identity/v3/projects/68b2ebaa92044f5b96e0e6ca1d3c99a2/users/b563c1dc9f8943669879e62c10a23854/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:34.315581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4875dd27-0213-4ee3-9250-2f37cafcc729 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:34.315868 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4875dd27-0213-4ee3-9250-2f37cafcc729 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:34.316125 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4875dd27-0213-4ee3-9250-2f37cafcc729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:34.343132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-4875dd27-0213-4ee3-9250-2f37cafcc729 None None] MFA Rules not processed for user `b563c1dc9f8943669879e62c10a23854`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:34.344360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4875dd27-0213-4ee3-9250-2f37cafcc729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:34.373840 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:34.406487 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1378/2773] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:34.418084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:34.419772 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:34.455256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:34.477502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1079df-3065-4f6f-b39d-349699c9ab1f admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:34.481147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:34.481778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:34.482059 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:34.482332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:34.490921 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:34.492334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:34.501441 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1396/2774] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:34 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:38.673557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4875dd27-0213-4ee3-9250-2f37cafcc729 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:38.675143 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4875dd27-0213-4ee3-9250-2f37cafcc729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:38.748808 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4875dd27-0213-4ee3-9250-2f37cafcc729 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:38.754365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c366ed8d-5e44-4902-87be-c9f255f0553d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:38.754981 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c366ed8d-5e44-4902-87be-c9f255f0553d service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:38.755254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c366ed8d-5e44-4902-87be-c9f255f0553d service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:38.755526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c366ed8d-5e44-4902-87be-c9f255f0553d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:38.757260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c366ed8d-5e44-4902-87be-c9f255f0553d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:38.770019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c366ed8d-5e44-4902-87be-c9f255f0553d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:38.771633 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c366ed8d-5e44-4902-87be-c9f255f0553d service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:38.782248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1379/2775] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:44.738386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=12 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:48.777955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=47 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:53.854281 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:53.856206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:53.920258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8319be-3401-491e-87d7-15aa2813c9fd service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:53.924922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:53.925542 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:53.925850 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:53.926123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] PATH_INFO: `/v3/users/d9027c2734634603a7c8b4990b790300` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:53.927254 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9027c2734634603a7c8b4990b790300)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:53.929157 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:54.135069 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9027c2734634603a7c8b4990b790300'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:54.135724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9027c2734634603a7c8b4990b790300'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:54.148382 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9027c2734634603a7c8b4990b790300'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:54.221608 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9027c2734634603a7c8b4990b790300'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:54.229541 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9027c2734634603a7c8b4990b790300'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:54.235761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1397/2776] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:25:53 2022] DELETE /identity/v3/users/d9027c2734634603a7c8b4990b790300 => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:54.240335 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f2c318-54d5-488c-803b-4a646e68c52b None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:54.240778 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f2c318-54d5-488c-803b-4a646e68c52b None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:54.240893 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f2c318-54d5-488c-803b-4a646e68c52b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:54.259933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e4f2c318-54d5-488c-803b-4a646e68c52b None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:54.261130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e4f2c318-54d5-488c-803b-4a646e68c52b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:54.322168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1380/2777] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:54.336806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:54.338256 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:54.404423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8af735-d180-403a-ad10-808a9d67e122 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:54.408123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f76331-87b9-4b62-a83a-d11a48d65e07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:54.408827 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f76331-87b9-4b62-a83a-d11a48d65e07 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:54.409176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f76331-87b9-4b62-a83a-d11a48d65e07 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:54.409552 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f76331-87b9-4b62-a83a-d11a48d65e07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:54.420197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f76331-87b9-4b62-a83a-d11a48d65e07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:54.421954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f76331-87b9-4b62-a83a-d11a48d65e07 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:54.433530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1398/2778] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:25:54.911339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f2c318-54d5-488c-803b-4a646e68c52b None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:54.912834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e4f2c318-54d5-488c-803b-4a646e68c52b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:54.929736 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f2c318-54d5-488c-803b-4a646e68c52b None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:54.935646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:54.936307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:54.936717 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:54.937160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] PATH_INFO: `/v3/projects/bc9eef4ecfd04739beac5344bcec4271` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:54.938352 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc9eef4ecfd04739beac5344bcec4271)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:54.941000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:55.053319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc9eef4ecfd04739beac5344bcec4271', 'request_id': 'req-d1322050-0b03-4126-9a49-813f0154f996'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:55.172046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc9eef4ecfd04739beac5344bcec4271', 'request_id': 'req-d1322050-0b03-4126-9a49-813f0154f996'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:55.178197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc9eef4ecfd04739beac5344bcec4271', 'request_id': 'req-d1322050-0b03-4126-9a49-813f0154f996'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:55.189314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] The token cache is being invalidate because project bc9eef4ecfd04739beac5344bcec4271 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:25:55.189625 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:55.190270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-d1322050-0b03-4126-9a49-813f0154f996 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:25:55.191757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1381/2779] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:25:54 2022] DELETE /identity/v3/projects/bc9eef4ecfd04739beac5344bcec4271 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:55.196619 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6c2757-d715-453a-8dda-129cf628ea83 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.197090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6c2757-d715-453a-8dda-129cf628ea83 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.197193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6c2757-d715-453a-8dda-129cf628ea83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.216714 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0a6c2757-d715-453a-8dda-129cf628ea83 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:55.217962 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0a6c2757-d715-453a-8dda-129cf628ea83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:55.237909 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:55.279205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1399/2780] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:55.342628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0b862-43ba-41d8-b7aa-7f97d1005ee3 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.343303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0b862-43ba-41d8-b7aa-7f97d1005ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.343303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc0b862-43ba-41d8-b7aa-7f97d1005ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.363471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-ffc0b862-43ba-41d8-b7aa-7f97d1005ee3 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:55.368657 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ffc0b862-43ba-41d8-b7aa-7f97d1005ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:55.372602 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1382/2781] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:55.376961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6c2757-d715-453a-8dda-129cf628ea83 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:55.386640 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6c2757-d715-453a-8dda-129cf628ea83 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:55.391750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:55.392342 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.392621 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.392892 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.393529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:25:55.394359 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:55.395944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:55.407293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1400/2782] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:25:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:25:55.412452 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc0b862-43ba-41d8-b7aa-7f97d1005ee3 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:55.421707 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc0b862-43ba-41d8-b7aa-7f97d1005ee3 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:55.426270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52359978-675e-4709-b49d-d3da60dc76a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:55.426864 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52359978-675e-4709-b49d-d3da60dc76a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.427145 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52359978-675e-4709-b49d-d3da60dc76a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.427415 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52359978-675e-4709-b49d-d3da60dc76a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.428035 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52359978-675e-4709-b49d-d3da60dc76a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:55.429978 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52359978-675e-4709-b49d-d3da60dc76a8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:55.516585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1383/2783] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:25:55 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:25:55.520643 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:55.531222 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e9b088-c64d-4a60-8041-a4e62465569d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:55.588744 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb07380f-e76a-4932-b3f2-7258976a128d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:55.589345 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb07380f-e76a-4932-b3f2-7258976a128d admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.589652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb07380f-e76a-4932-b3f2-7258976a128d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.589928 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb07380f-e76a-4932-b3f2-7258976a128d admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.590553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb07380f-e76a-4932-b3f2-7258976a128d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:55.592448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb07380f-e76a-4932-b3f2-7258976a128d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:55.703211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1401/2784] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:25:55 2022] POST /identity/v3/users => generated 293 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:25:55.707211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52359978-675e-4709-b49d-d3da60dc76a8 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:55.718264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52359978-675e-4709-b49d-d3da60dc76a8 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:55.722894 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbcc535-11b1-4628-aa66-5a3386a9868b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:55.723505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbcc535-11b1-4628-aa66-5a3386a9868b admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.724032 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbcc535-11b1-4628-aa66-5a3386a9868b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.724303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbcc535-11b1-4628-aa66-5a3386a9868b admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.724985 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbcc535-11b1-4628-aa66-5a3386a9868b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:55.726611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbcc535-11b1-4628-aa66-5a3386a9868b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:55.736882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1384/2785] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:55.740907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb07380f-e76a-4932-b3f2-7258976a128d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:55.750841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb07380f-e76a-4932-b3f2-7258976a128d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:55.756042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67e60b9-81a6-4330-813b-821eb5eac7a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:55.756629 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e60b9-81a6-4330-813b-821eb5eac7a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.756906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e60b9-81a6-4330-813b-821eb5eac7a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.757176 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e60b9-81a6-4330-813b-821eb5eac7a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.757859 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67e60b9-81a6-4330-813b-821eb5eac7a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:55.759440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67e60b9-81a6-4330-813b-821eb5eac7a9 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:55.769295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1402/2786] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:25:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:25:55.773897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbcc535-11b1-4628-aa66-5a3386a9868b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:55.783836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbcc535-11b1-4628-aa66-5a3386a9868b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:55.788472 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:55.789120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.789400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.789712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] PATH_INFO: `/v3/projects/fc21ab560b1740b8893d3fd4ff92798b/users/093dd3cd8b4842bebb60027e72675fd0/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.804747 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc21ab560b1740b8893d3fd4ff92798b, user_id=093dd3cd8b4842bebb60027e72675fd0, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:55.807425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:55.903283 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1385/2787] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:25:55 2022] PUT /identity/v3/projects/fc21ab560b1740b8893d3fd4ff92798b/users/093dd3cd8b4842bebb60027e72675fd0/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:25:55.913395 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e7e5d-329b-4dcb-9647-e1bf80f34881 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:55.913717 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e7e5d-329b-4dcb-9647-e1bf80f34881 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:55.913996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10e7e5d-329b-4dcb-9647-e1bf80f34881 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:55.941386 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-c10e7e5d-329b-4dcb-9647-e1bf80f34881 None None] MFA Rules not processed for user `093dd3cd8b4842bebb60027e72675fd0`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:25:55.943327 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c10e7e5d-329b-4dcb-9647-e1bf80f34881 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:56.013876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1403/2788] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:25:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:25:56.024999 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:25:56.026814 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:25:56.048260 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:25:56.083911 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31308b-b2df-40f0-a829-42b23edba7b5 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:25:56.087533 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7c3d08-1584-43da-9a61-67dafe2020db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:25:56.088151 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c3d08-1584-43da-9a61-67dafe2020db service cinder] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:25:56.088488 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c3d08-1584-43da-9a61-67dafe2020db service cinder] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:25:56.088769 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c3d08-1584-43da-9a61-67dafe2020db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:25:56.096843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7c3d08-1584-43da-9a61-67dafe2020db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:25:56.098314 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7c3d08-1584-43da-9a61-67dafe2020db service cinder] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:25:56.107242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1386/2789] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:25:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:26:06.011032 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:06.324725 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=17 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:21.962316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10e7e5d-329b-4dcb-9647-e1bf80f34881 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:22.028080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10e7e5d-329b-4dcb-9647-e1bf80f34881 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:22.032948 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:22.033581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:22.033885 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:22.034156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] PATH_INFO: `/v3/users/093dd3cd8b4842bebb60027e72675fd0` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:22.035293 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=093dd3cd8b4842bebb60027e72675fd0)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:22.037205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:22.309829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '093dd3cd8b4842bebb60027e72675fd0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:22.310479 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '093dd3cd8b4842bebb60027e72675fd0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:22.322905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '093dd3cd8b4842bebb60027e72675fd0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:22.428596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '093dd3cd8b4842bebb60027e72675fd0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:22.435807 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '093dd3cd8b4842bebb60027e72675fd0'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:22.441829 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1404/2790] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:26:21 2022] DELETE /identity/v3/users/093dd3cd8b4842bebb60027e72675fd0 => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:26:22.446892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db80388-e39d-4965-a6d7-cd8c163e2010 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:22.455890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db80388-e39d-4965-a6d7-cd8c163e2010 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:22.455890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db80388-e39d-4965-a6d7-cd8c163e2010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:22.467216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-5db80388-e39d-4965-a6d7-cd8c163e2010 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:22.468128 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5db80388-e39d-4965-a6d7-cd8c163e2010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:22.528620 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1387/2791] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:22.538213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:22.539596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:22.612245 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be918ee6-e348-45d3-8868-f84fd9c659d6 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:22.616008 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a17f21-ea6c-49c4-b55b-77026ba43316 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:22.616703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a17f21-ea6c-49c4-b55b-77026ba43316 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:22.617046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a17f21-ea6c-49c4-b55b-77026ba43316 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:22.617486 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a17f21-ea6c-49c4-b55b-77026ba43316 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:22.629078 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a17f21-ea6c-49c4-b55b-77026ba43316 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:22.630525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a17f21-ea6c-49c4-b55b-77026ba43316 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:22.640692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1405/2792] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:26:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:26:23.177348 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db80388-e39d-4965-a6d7-cd8c163e2010 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:23.178811 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5db80388-e39d-4965-a6d7-cd8c163e2010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:23.191057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db80388-e39d-4965-a6d7-cd8c163e2010 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:23.196381 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:23.197020 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:23.197300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:23.197577 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] PATH_INFO: `/v3/projects/fc21ab560b1740b8893d3fd4ff92798b` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:23.200040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc21ab560b1740b8893d3fd4ff92798b)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:23.202320 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:23.343418 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc21ab560b1740b8893d3fd4ff92798b', 'request_id': 'req-1f1bde94-fe98-4662-8956-e7409079fa58'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:23.411468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc21ab560b1740b8893d3fd4ff92798b', 'request_id': 'req-1f1bde94-fe98-4662-8956-e7409079fa58'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:23.417972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc21ab560b1740b8893d3fd4ff92798b', 'request_id': 'req-1f1bde94-fe98-4662-8956-e7409079fa58'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:23.429655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] The token cache is being invalidate because project fc21ab560b1740b8893d3fd4ff92798b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:26:23.429897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:23.430539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-1f1bde94-fe98-4662-8956-e7409079fa58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:23.432036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1388/2793] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:26:23 2022] DELETE /identity/v3/projects/fc21ab560b1740b8893d3fd4ff92798b => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:26:23.437043 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ea436-577f-4969-8587-e8c7d9af1c5c None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:23.437550 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ea436-577f-4969-8587-e8c7d9af1c5c None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:23.437816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ea436-577f-4969-8587-e8c7d9af1c5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:23.457038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-344ea436-577f-4969-8587-e8c7d9af1c5c None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:23.458315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-344ea436-577f-4969-8587-e8c7d9af1c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:23.476763 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:23.517571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1406/2794] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:23.527439 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cba802-bd14-4814-84f2-d2c645e0f09d None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:23.527933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cba802-bd14-4814-84f2-d2c645e0f09d None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:23.528058 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cba802-bd14-4814-84f2-d2c645e0f09d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:23.547773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c6cba802-bd14-4814-84f2-d2c645e0f09d None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:23.548962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c6cba802-bd14-4814-84f2-d2c645e0f09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:23.557016 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1389/2795] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:23.561408 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344ea436-577f-4969-8587-e8c7d9af1c5c None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:23.571107 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344ea436-577f-4969-8587-e8c7d9af1c5c None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:23.575821 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:23.577168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:23.577168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:23.577168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:23.577613 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:26:23.577940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:23.579520 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:23.591316 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1407/2796] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:26:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:26:23.595642 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cba802-bd14-4814-84f2-d2c645e0f09d None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:23.605639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cba802-bd14-4814-84f2-d2c645e0f09d None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:23.610173 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832cfd85-5299-461c-a0da-4284eeba11ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:23.610763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832cfd85-5299-461c-a0da-4284eeba11ca admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:23.611037 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832cfd85-5299-461c-a0da-4284eeba11ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:23.611303 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832cfd85-5299-461c-a0da-4284eeba11ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:23.611931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832cfd85-5299-461c-a0da-4284eeba11ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:23.613827 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832cfd85-5299-461c-a0da-4284eeba11ca admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:23.722116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1390/2797] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:26:23 2022] POST /identity/v3/projects => generated 367 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:26:23.727100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:23.736842 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c36211-bc1c-4a28-bde3-2034cc1b67a4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:23.792121 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66036cae-e34f-4b68-9bee-37341b3f36bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:23.792721 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66036cae-e34f-4b68-9bee-37341b3f36bf admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:23.793001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66036cae-e34f-4b68-9bee-37341b3f36bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:23.793273 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66036cae-e34f-4b68-9bee-37341b3f36bf admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:23.793922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66036cae-e34f-4b68-9bee-37341b3f36bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:23.795840 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66036cae-e34f-4b68-9bee-37341b3f36bf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:23.932416 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1408/2798] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:26:23 2022] POST /identity/v3/users => generated 284 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:26:23.936349 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832cfd85-5299-461c-a0da-4284eeba11ca admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:23.946875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832cfd85-5299-461c-a0da-4284eeba11ca admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:23.951432 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb706fc-ff1c-44bb-bf1a-ca759f99a1ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:23.951962 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb706fc-ff1c-44bb-bf1a-ca759f99a1ca admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:23.952242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb706fc-ff1c-44bb-bf1a-ca759f99a1ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:23.952512 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb706fc-ff1c-44bb-bf1a-ca759f99a1ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:23.953157 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb706fc-ff1c-44bb-bf1a-ca759f99a1ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:23.957423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb706fc-ff1c-44bb-bf1a-ca759f99a1ca admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:23.968882 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1391/2799] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:26:23.970392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66036cae-e34f-4b68-9bee-37341b3f36bf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:23.981180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66036cae-e34f-4b68-9bee-37341b3f36bf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:23.985159 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6669b84-0820-4010-badb-a122f6cd5325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:23.985816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6669b84-0820-4010-badb-a122f6cd5325 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:23.986097 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6669b84-0820-4010-badb-a122f6cd5325 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:23.988053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6669b84-0820-4010-badb-a122f6cd5325 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:23.988053 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6669b84-0820-4010-badb-a122f6cd5325 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:23.988548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6669b84-0820-4010-badb-a122f6cd5325 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:23.999332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1409/2800] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:26:24.003526 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb706fc-ff1c-44bb-bf1a-ca759f99a1ca admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:24.013940 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb706fc-ff1c-44bb-bf1a-ca759f99a1ca admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:24.018361 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:24.019003 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:24.019279 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:24.019546 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] PATH_INFO: `/v3/projects/cac1cbc4cf3547288a7e5f6edea8e4c8/users/3760151f95ce4c6c88f24b51ff3b6625/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:24.034905 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] RBAC: Authorizing `identity:create_grant(project_id=cac1cbc4cf3547288a7e5f6edea8e4c8, user_id=3760151f95ce4c6c88f24b51ff3b6625, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:24.037545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:24.126000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1392/2801] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:26:24 2022] PUT /identity/v3/projects/cac1cbc4cf3547288a7e5f6edea8e4c8/users/3760151f95ce4c6c88f24b51ff3b6625/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:26:24.140544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085e7b85-2bae-42ae-95d4-c5f218ef9a3a None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:24.140831 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085e7b85-2bae-42ae-95d4-c5f218ef9a3a None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:24.141106 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085e7b85-2bae-42ae-95d4-c5f218ef9a3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:24.151950 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:24.153741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:24.168873 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-085e7b85-2bae-42ae-95d4-c5f218ef9a3a None None] MFA Rules not processed for user `3760151f95ce4c6c88f24b51ff3b6625`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:24.170090 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-085e7b85-2bae-42ae-95d4-c5f218ef9a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:24.175558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:24.218383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961c4a77-723c-4c12-b813-e62789af8fcc admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:24.223629 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:24.224549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:24.224832 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:24.225104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] PATH_INFO: `/v3/users/b563c1dc9f8943669879e62c10a23854` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:24.226391 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=b563c1dc9f8943669879e62c10a23854)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:24.228329 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:24.242274 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1410/2802] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:24 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:24.253651 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085e7b85-2bae-42ae-95d4-c5f218ef9a3a None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:24.255042 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-085e7b85-2bae-42ae-95d4-c5f218ef9a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:24.275695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:24.312691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085e7b85-2bae-42ae-95d4-c5f218ef9a3a None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:24.316320 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcae1e2f-bcc9-42c7-94df-3e8bf0f27507 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6cc8f34f9674a94ba23a7157e44a6f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:24.316925 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcae1e2f-bcc9-42c7-94df-3e8bf0f27507 service glance] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:24.317205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcae1e2f-bcc9-42c7-94df-3e8bf0f27507 service glance] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:24.317518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcae1e2f-bcc9-42c7-94df-3e8bf0f27507 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:24.326420 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcae1e2f-bcc9-42c7-94df-3e8bf0f27507 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:24.327838 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcae1e2f-bcc9-42c7-94df-3e8bf0f27507 service glance] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:24.336648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1411/2803] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:26:24 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:26:24.452918 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b563c1dc9f8943669879e62c10a23854'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:24.464798 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b563c1dc9f8943669879e62c10a23854'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:24.465529 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b563c1dc9f8943669879e62c10a23854'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:24.529619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b563c1dc9f8943669879e62c10a23854'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:24.537301 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b563c1dc9f8943669879e62c10a23854'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:24.544055 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1393/2804] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:26:24 2022] DELETE /identity/v3/users/b563c1dc9f8943669879e62c10a23854 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:26:24.548709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db1ef0-93c0-43f4-a8d3-007e01de6986 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:24.549168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db1ef0-93c0-43f4-a8d3-007e01de6986 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:24.549392 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db1ef0-93c0-43f4-a8d3-007e01de6986 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:24.568954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-41db1ef0-93c0-43f4-a8d3-007e01de6986 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:24.570250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-41db1ef0-93c0-43f4-a8d3-007e01de6986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:24.633538 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1412/2805] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:24.643569 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:24.645001 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:24.702611 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd41a9-6c47-4c38-9b29-57dfce70eb1b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:24.706434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9869ff73-7f3e-4863-ad57-22c3f1890225 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:24.707867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9869ff73-7f3e-4863-ad57-22c3f1890225 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:24.707867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9869ff73-7f3e-4863-ad57-22c3f1890225 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:24.707867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9869ff73-7f3e-4863-ad57-22c3f1890225 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:24.717410 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9869ff73-7f3e-4863-ad57-22c3f1890225 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:24.718906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9869ff73-7f3e-4863-ad57-22c3f1890225 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:24.729072 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1394/2806] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:26:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:26:25.161738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41db1ef0-93c0-43f4-a8d3-007e01de6986 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:25.163927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-41db1ef0-93c0-43f4-a8d3-007e01de6986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:25.190330 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41db1ef0-93c0-43f4-a8d3-007e01de6986 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:25.195631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:25.196250 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:25.196559 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:25.196835 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] PATH_INFO: `/v3/projects/68b2ebaa92044f5b96e0e6ca1d3c99a2` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:25.198134 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] RBAC: Authorizing `identity:delete_project(project_id=68b2ebaa92044f5b96e0e6ca1d3c99a2)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:25.200087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:25.308080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68b2ebaa92044f5b96e0e6ca1d3c99a2', 'request_id': 'req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:25.314177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68b2ebaa92044f5b96e0e6ca1d3c99a2', 'request_id': 'req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:25.410119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68b2ebaa92044f5b96e0e6ca1d3c99a2', 'request_id': 'req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:25.423999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] The token cache is being invalidate because project 68b2ebaa92044f5b96e0e6ca1d3c99a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:26:25.424509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:25.425287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-61283fb3-b7f2-47ea-97ae-600dea4b6f5b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:26:25.427462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1413/2807] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:26:25 2022] DELETE /identity/v3/projects/68b2ebaa92044f5b96e0e6ca1d3c99a2 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:26:25.432457 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec663d-acb3-4c06-8f1c-820abbda74c1 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:25.432751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec663d-acb3-4c06-8f1c-820abbda74c1 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:25.433049 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec663d-acb3-4c06-8f1c-820abbda74c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:25.455325 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c9ec663d-acb3-4c06-8f1c-820abbda74c1 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:25.456619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c9ec663d-acb3-4c06-8f1c-820abbda74c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:25.495828 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:25.517728 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1395/2808] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:25.527607 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3629a927-a1ac-4df4-b772-42c6821f3a07 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:25.527902 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3629a927-a1ac-4df4-b772-42c6821f3a07 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:25.528199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3629a927-a1ac-4df4-b772-42c6821f3a07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:25.550410 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-3629a927-a1ac-4df4-b772-42c6821f3a07 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:25.551645 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3629a927-a1ac-4df4-b772-42c6821f3a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:25.560068 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1414/2809] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:25.564132 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec663d-acb3-4c06-8f1c-820abbda74c1 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:25.574904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec663d-acb3-4c06-8f1c-820abbda74c1 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:25.580041 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:25.580492 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:25.580780 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:25.581059 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] PATH_INFO: `/v3/domains` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:25.581972 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81763) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:26:25.582284 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:25.584004 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:25.599447 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1396/2810] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:26:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:26:25.603841 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3629a927-a1ac-4df4-b772-42c6821f3a07 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:25.613793 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3629a927-a1ac-4df4-b772-42c6821f3a07 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:25.618824 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80477365-b27d-4296-8b7e-c308c974b19d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:25.619428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80477365-b27d-4296-8b7e-c308c974b19d admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:25.619706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80477365-b27d-4296-8b7e-c308c974b19d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:25.619988 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80477365-b27d-4296-8b7e-c308c974b19d admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:25.620615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80477365-b27d-4296-8b7e-c308c974b19d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:25.622540 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80477365-b27d-4296-8b7e-c308c974b19d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:25.733350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1415/2811] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:26:25 2022] POST /identity/v3/projects => generated 385 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:26:25.737539 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:25.748605 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791a6e3a-6634-4e3c-8662-06223cee34fb admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:25.811634 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83589d9-6053-4b15-bf3a-2a2ec0c1c18c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:25.812197 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83589d9-6053-4b15-bf3a-2a2ec0c1c18c admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:25.812470 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83589d9-6053-4b15-bf3a-2a2ec0c1c18c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:25.812739 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83589d9-6053-4b15-bf3a-2a2ec0c1c18c admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:25.813405 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83589d9-6053-4b15-bf3a-2a2ec0c1c18c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:25.815368 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83589d9-6053-4b15-bf3a-2a2ec0c1c18c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:25.903545 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1397/2812] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:26:25 2022] POST /identity/v3/users => generated 293 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:26:25.907381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80477365-b27d-4296-8b7e-c308c974b19d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:25.917728 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80477365-b27d-4296-8b7e-c308c974b19d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:25.922748 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8f433c-36d2-4fea-81c3-171527169194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:25.923365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f433c-36d2-4fea-81c3-171527169194 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:25.923671 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f433c-36d2-4fea-81c3-171527169194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:25.923940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8f433c-36d2-4fea-81c3-171527169194 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:25.924585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8f433c-36d2-4fea-81c3-171527169194 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:25.926218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8f433c-36d2-4fea-81c3-171527169194 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:25.937091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1416/2813] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:26:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:26:25.941913 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83589d9-6053-4b15-bf3a-2a2ec0c1c18c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:25.952597 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83589d9-6053-4b15-bf3a-2a2ec0c1c18c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:25.958592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c720dfa2-6ad9-4353-98f1-5987691e20f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:25.958592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720dfa2-6ad9-4353-98f1-5987691e20f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:25.958592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720dfa2-6ad9-4353-98f1-5987691e20f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:25.958592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720dfa2-6ad9-4353-98f1-5987691e20f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:25.959079 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c720dfa2-6ad9-4353-98f1-5987691e20f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:25.960686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c720dfa2-6ad9-4353-98f1-5987691e20f3 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:25.973138 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1398/2814] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:26:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:26:25.977926 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8f433c-36d2-4fea-81c3-171527169194 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:25.984440 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:25.994334 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8f433c-36d2-4fea-81c3-171527169194 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:25.999138 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:25.999794 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:26.000181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:26.000458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] PATH_INFO: `/v3/projects/b0f35b076ec94efda11070e02d3a5acd/users/fb70879694d443df9b30ff172cfa39b4/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:26.018904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0f35b076ec94efda11070e02d3a5acd, user_id=fb70879694d443df9b30ff172cfa39b4, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:26.022295 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:26.138233 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1417/2815] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:26:25 2022] PUT /identity/v3/projects/b0f35b076ec94efda11070e02d3a5acd/users/fb70879694d443df9b30ff172cfa39b4/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:26:26.143683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e1f19a-53e4-4745-a1d3-e1ee887944b9 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:26.143973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e1f19a-53e4-4745-a1d3-e1ee887944b9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:26.144292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e1f19a-53e4-4745-a1d3-e1ee887944b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:26.167636 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-51e1f19a-53e4-4745-a1d3-e1ee887944b9 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:26.168852 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-51e1f19a-53e4-4745-a1d3-e1ee887944b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:26.235357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1399/2816] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:26:26.244922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:26.246740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:26.303808 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b20725-52ed-4516-9e26-1afb145dba12 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:26.307548 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443a6b16-58a1-413b-8041-63c33d3c3ec8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:26.308148 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443a6b16-58a1-413b-8041-63c33d3c3ec8 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:26.308428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443a6b16-58a1-413b-8041-63c33d3c3ec8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:26.308701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443a6b16-58a1-413b-8041-63c33d3c3ec8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:26.317477 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443a6b16-58a1-413b-8041-63c33d3c3ec8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:26.319444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443a6b16-58a1-413b-8041-63c33d3c3ec8 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:26.328771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1418/2817] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:26:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:26:32.518354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b10c19-cacf-4627-a7a1-17f78fee2756 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:32.518354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b10c19-cacf-4627-a7a1-17f78fee2756 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:32.518354 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b10c19-cacf-4627-a7a1-17f78fee2756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:32.549266 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-d7b10c19-cacf-4627-a7a1-17f78fee2756 None None] MFA Rules not processed for user `fb70879694d443df9b30ff172cfa39b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:32.550651 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d7b10c19-cacf-4627-a7a1-17f78fee2756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:32.592854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:32.620763 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1400/2818] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:32.625492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443a6b16-58a1-413b-8041-63c33d3c3ec8 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:32.636028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443a6b16-58a1-413b-8041-63c33d3c3ec8 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:32.640766 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b504b6-bd86-4e4e-93b7-721b1d4f963a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:32.641363 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b504b6-bd86-4e4e-93b7-721b1d4f963a admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:32.641669 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b504b6-bd86-4e4e-93b7-721b1d4f963a admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:32.641947 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b504b6-bd86-4e4e-93b7-721b1d4f963a admin admin] PATH_INFO: `/v3/projects` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:32.642686 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b504b6-bd86-4e4e-93b7-721b1d4f963a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:32.644581 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b504b6-bd86-4e4e-93b7-721b1d4f963a admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:32.800907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1419/2819] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:26:32 2022] POST /identity/v3/projects => generated 383 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:26:32.805373 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b10c19-cacf-4627-a7a1-17f78fee2756 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:32.817971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b10c19-cacf-4627-a7a1-17f78fee2756 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:32.885365 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306550f-0543-4901-9aac-bd6fc3ec0507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:32.886017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306550f-0543-4901-9aac-bd6fc3ec0507 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:32.886300 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306550f-0543-4901-9aac-bd6fc3ec0507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:32.886764 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306550f-0543-4901-9aac-bd6fc3ec0507 admin admin] PATH_INFO: `/v3/users` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:32.888093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306550f-0543-4901-9aac-bd6fc3ec0507 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:32.890080 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306550f-0543-4901-9aac-bd6fc3ec0507 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:33.007393 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1401/2820] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:26:32 2022] POST /identity/v3/users => generated 298 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:26:33.011639 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b504b6-bd86-4e4e-93b7-721b1d4f963a admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:33.022478 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b504b6-bd86-4e4e-93b7-721b1d4f963a admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:33.027289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dce25a3-894d-4c68-8b5a-b6fd13ff9685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:33.027895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce25a3-894d-4c68-8b5a-b6fd13ff9685 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:33.028171 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce25a3-894d-4c68-8b5a-b6fd13ff9685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:33.028442 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce25a3-894d-4c68-8b5a-b6fd13ff9685 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:33.029112 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dce25a3-894d-4c68-8b5a-b6fd13ff9685 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:33.030740 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dce25a3-894d-4c68-8b5a-b6fd13ff9685 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:33.044204 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1420/2821] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:26:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:26:33.048551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306550f-0543-4901-9aac-bd6fc3ec0507 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:33.060498 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306550f-0543-4901-9aac-bd6fc3ec0507 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:33.066160 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9576e5d0-8fc4-4e72-a28c-7a67dbc8861b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:33.066839 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9576e5d0-8fc4-4e72-a28c-7a67dbc8861b admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:33.067120 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9576e5d0-8fc4-4e72-a28c-7a67dbc8861b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:33.073400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9576e5d0-8fc4-4e72-a28c-7a67dbc8861b admin admin] PATH_INFO: `/v3/projects/52469a83e4f74020b1208ed6ab573320/users/4f4a3d3b68dc481c8e4a099bde99d686/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:33.083749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9576e5d0-8fc4-4e72-a28c-7a67dbc8861b admin admin] RBAC: Authorizing `identity:create_grant(project_id=52469a83e4f74020b1208ed6ab573320, user_id=4f4a3d3b68dc481c8e4a099bde99d686, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:33.086499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9576e5d0-8fc4-4e72-a28c-7a67dbc8861b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:33.177528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1402/2822] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:26:33 2022] PUT /identity/v3/projects/52469a83e4f74020b1208ed6ab573320/users/4f4a3d3b68dc481c8e4a099bde99d686/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:26:33.182446 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dce25a3-894d-4c68-8b5a-b6fd13ff9685 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:33.194466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dce25a3-894d-4c68-8b5a-b6fd13ff9685 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:33.262931 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f312604-149d-4177-81a5-f11b226fefc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:33.263582 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f312604-149d-4177-81a5-f11b226fefc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:33.263866 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f312604-149d-4177-81a5-f11b226fefc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:33.264252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f312604-149d-4177-81a5-f11b226fefc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:33.264790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f312604-149d-4177-81a5-f11b226fefc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:33.266476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f312604-149d-4177-81a5-f11b226fefc2 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:33.278779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1421/2823] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:26:33 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:26:33.283342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9576e5d0-8fc4-4e72-a28c-7a67dbc8861b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:33.295398 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9576e5d0-8fc4-4e72-a28c-7a67dbc8861b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:33.301426 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:33.302295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:33.308043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:33.308043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] PATH_INFO: `/v3/projects/52469a83e4f74020b1208ed6ab573320/users/4f4a3d3b68dc481c8e4a099bde99d686/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:33.308043 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52469a83e4f74020b1208ed6ab573320, user_id=4f4a3d3b68dc481c8e4a099bde99d686, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:33.308532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:33.325712 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1403/2824] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:26:33 2022] PUT /identity/v3/projects/52469a83e4f74020b1208ed6ab573320/users/4f4a3d3b68dc481c8e4a099bde99d686/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:26:39.322784 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f906515-d068-4190-957f-6bcb5f32217f None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:39.323309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f906515-d068-4190-957f-6bcb5f32217f None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:39.323309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f906515-d068-4190-957f-6bcb5f32217f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:39.354667 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-0f906515-d068-4190-957f-6bcb5f32217f None None] MFA Rules not processed for user `4f4a3d3b68dc481c8e4a099bde99d686`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:39.355893 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-0f906515-d068-4190-957f-6bcb5f32217f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:39.389537 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:39.422904 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1422/2825] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:39 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:39.435504 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:39.437240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:39.499784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afdc3f8-aa93-4ab8-a82e-16ba1617e7e7 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:39.503358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a6cc8f34f9674a94ba23a7157e44a6f2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:39.503966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:39.504240 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:39.504506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:39.548022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:39.560915 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:39.562377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:39.571044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1404/2826] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:26:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:26:39.852499 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1607cdce-f0e2-47fc-9de8-f4aeee3433a0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:39.852882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1607cdce-f0e2-47fc-9de8-f4aeee3433a0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:39.853218 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1607cdce-f0e2-47fc-9de8-f4aeee3433a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:39.904692 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-1607cdce-f0e2-47fc-9de8-f4aeee3433a0 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:39.906087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1607cdce-f0e2-47fc-9de8-f4aeee3433a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:39.972724 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1423/2827] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:26:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:40.179941 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:40.181669 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:40.275152 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c2cb2-8f80-4fd2-a3ac-8f7806c2aa6d service glance] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:40.281430 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326d8a0e-6521-43fb-a942-5ec1a215f879 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:40.282091 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326d8a0e-6521-43fb-a942-5ec1a215f879 service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:40.282372 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326d8a0e-6521-43fb-a942-5ec1a215f879 service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:40.282647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326d8a0e-6521-43fb-a942-5ec1a215f879 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:40.293044 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326d8a0e-6521-43fb-a942-5ec1a215f879 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:40.294781 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326d8a0e-6521-43fb-a942-5ec1a215f879 service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:40.303749 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1405/2828] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:26:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:26:49.969409 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=53 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:50.299715 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=48 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:26:50.831381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb202871-dc09-4191-b703-bf9df7f3811d None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:50.832143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb202871-dc09-4191-b703-bf9df7f3811d None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:50.832143 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb202871-dc09-4191-b703-bf9df7f3811d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:50.851209 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-eb202871-dc09-4191-b703-bf9df7f3811d None None] MFA Rules not processed for user `fb70879694d443df9b30ff172cfa39b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:26:50.852401 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-eb202871-dc09-4191-b703-bf9df7f3811d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:50.859210 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1424/2829] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:26:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:26:51.112853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326d8a0e-6521-43fb-a942-5ec1a215f879 service swift] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:26:51.165787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326d8a0e-6521-43fb-a942-5ec1a215f879 service swift] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:26:51.169359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:26:51.169983 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:26:51.170259 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:26:51.170524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:26:51.172225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:26:51.182248 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:26:51.183685 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:26:51.191440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1406/2830] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:26:51 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:27:00.845792 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:01.187907 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=26 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:02.913428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea3068c-8bcb-40be-ba03-dbe2fdba4649 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:02.914296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea3068c-8bcb-40be-ba03-dbe2fdba4649 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:02.914296 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea3068c-8bcb-40be-ba03-dbe2fdba4649 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:02.934980 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-bea3068c-8bcb-40be-ba03-dbe2fdba4649 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:27:02.936192 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bea3068c-8bcb-40be-ba03-dbe2fdba4649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:02.943944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1425/2831] 10.222.0.16 () {58 vars in 1009 bytes} [Thu Aug 25 05:27:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:27:02.957416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:02.970582 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929a864f-2bbc-42e5-9a01-ef11d1dfb541 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:02.976297 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d276b4c-0f19-4988-9800-7dfbd7f46b5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:02.976897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d276b4c-0f19-4988-9800-7dfbd7f46b5d service swift] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:02.977182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d276b4c-0f19-4988-9800-7dfbd7f46b5d service swift] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:02.977454 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d276b4c-0f19-4988-9800-7dfbd7f46b5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:02.985708 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d276b4c-0f19-4988-9800-7dfbd7f46b5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:02.987280 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d276b4c-0f19-4988-9800-7dfbd7f46b5d service swift] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:02.995968 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1407/2832] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:27:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:27:12.930028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:12.992613 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:21.851404 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea3068c-8bcb-40be-ba03-dbe2fdba4649 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:21.852933 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bea3068c-8bcb-40be-ba03-dbe2fdba4649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:21.923673 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea3068c-8bcb-40be-ba03-dbe2fdba4649 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:21.928235 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:21.928949 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:21.929232 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:21.929509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] PATH_INFO: `/v3/users/6a4136b08cc1464eac6c779652a1d2b1` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:21.930751 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a4136b08cc1464eac6c779652a1d2b1)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:21.932731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:22.141674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a4136b08cc1464eac6c779652a1d2b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.142333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a4136b08cc1464eac6c779652a1d2b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.155137 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a4136b08cc1464eac6c779652a1d2b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.231128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a4136b08cc1464eac6c779652a1d2b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.238466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-5a0ae0e7-4767-496b-b647-19324d202614 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a4136b08cc1464eac6c779652a1d2b1'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.244372 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1426/2833] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:27:21 2022] DELETE /identity/v3/users/6a4136b08cc1464eac6c779652a1d2b1 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:27:22.480980 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d276b4c-0f19-4988-9800-7dfbd7f46b5d service swift] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:22.492100 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d276b4c-0f19-4988-9800-7dfbd7f46b5d service swift] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:22.556448 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:22.558315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:22.558315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:22.558315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] PATH_INFO: `/v3/projects/aed82d3b5299415ca1e2e3cfcb7e2d0b` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:22.558908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aed82d3b5299415ca1e2e3cfcb7e2d0b)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:22.560818 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:22.677093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aed82d3b5299415ca1e2e3cfcb7e2d0b', 'request_id': 'req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.785824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aed82d3b5299415ca1e2e3cfcb7e2d0b', 'request_id': 'req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.795369 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aed82d3b5299415ca1e2e3cfcb7e2d0b', 'request_id': 'req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.805933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] The token cache is being invalidate because project aed82d3b5299415ca1e2e3cfcb7e2d0b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:27:22.806295 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.806969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-08fa7fdc-6a75-4105-98c4-7f29b52f97f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:22.808462 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1408/2834] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:27:22 2022] DELETE /identity/v3/projects/aed82d3b5299415ca1e2e3cfcb7e2d0b => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:27:22.813996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb334e5-eeae-48fb-8253-9669245abf0e None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:22.814300 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb334e5-eeae-48fb-8253-9669245abf0e None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:22.814575 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb334e5-eeae-48fb-8253-9669245abf0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:22.835006 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-eeb334e5-eeae-48fb-8253-9669245abf0e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:27:22.836270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-eeb334e5-eeae-48fb-8253-9669245abf0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:22.906922 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1427/2835] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:27:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:27:22.916211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75463241-d85c-41c8-97da-1f4f71e3545e None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:22.916568 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75463241-d85c-41c8-97da-1f4f71e3545e None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:22.916908 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75463241-d85c-41c8-97da-1f4f71e3545e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:22.937343 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-75463241-d85c-41c8-97da-1f4f71e3545e None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:27:22.938637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-75463241-d85c-41c8-97da-1f4f71e3545e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:22.947598 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1409/2836] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:27:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:27:22.951990 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb334e5-eeae-48fb-8253-9669245abf0e None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:22.961944 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb334e5-eeae-48fb-8253-9669245abf0e None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:22.966734 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:22.967347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:22.967636 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:22.967916 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:22.968572 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:27:22.968876 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:22.970502 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:22.983631 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1428/2837] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:27:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 25 05:27:22.988017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75463241-d85c-41c8-97da-1f4f71e3545e None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:22.999336 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75463241-d85c-41c8-97da-1f4f71e3545e None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:23.004404 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2bfbfc-9ae9-43e1-b133-54ce3cce7f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:23.005017 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2bfbfc-9ae9-43e1-b133-54ce3cce7f89 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:23.010603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2bfbfc-9ae9-43e1-b133-54ce3cce7f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:23.010603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2bfbfc-9ae9-43e1-b133-54ce3cce7f89 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:23.010603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2bfbfc-9ae9-43e1-b133-54ce3cce7f89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:23.010603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2bfbfc-9ae9-43e1-b133-54ce3cce7f89 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:23.113012 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1410/2838] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:27:22 2022] POST /identity/v3/projects => generated 377 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:27:23.117128 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:23.128039 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be5d629-f307-4e52-9562-e4200b34a9e8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:23.131556 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:23.194102 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a5bf97-1f79-49b2-8c9e-3be194b206f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:23.195034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a5bf97-1f79-49b2-8c9e-3be194b206f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:23.195034 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a5bf97-1f79-49b2-8c9e-3be194b206f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:23.195298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a5bf97-1f79-49b2-8c9e-3be194b206f4 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:23.196326 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a5bf97-1f79-49b2-8c9e-3be194b206f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:23.198073 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a5bf97-1f79-49b2-8c9e-3be194b206f4 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:23.294935 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1429/2839] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:27:23 2022] POST /identity/v3/users => generated 289 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:27:23.298906 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2bfbfc-9ae9-43e1-b133-54ce3cce7f89 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:23.309889 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2bfbfc-9ae9-43e1-b133-54ce3cce7f89 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:23.314971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03daa20-8678-435c-852a-e2b779fa3d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:23.320853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03daa20-8678-435c-852a-e2b779fa3d79 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:23.320853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03daa20-8678-435c-852a-e2b779fa3d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:23.320853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03daa20-8678-435c-852a-e2b779fa3d79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:23.320853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03daa20-8678-435c-852a-e2b779fa3d79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:23.320853 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03daa20-8678-435c-852a-e2b779fa3d79 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:23.328836 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1411/2840] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:27:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:27:23.332940 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a5bf97-1f79-49b2-8c9e-3be194b206f4 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:23.343495 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a5bf97-1f79-49b2-8c9e-3be194b206f4 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:23.348761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30c63d5-6729-4584-b263-797d89678636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:23.349768 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c63d5-6729-4584-b263-797d89678636 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:23.350051 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c63d5-6729-4584-b263-797d89678636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:23.350354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30c63d5-6729-4584-b263-797d89678636 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:23.351003 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c63d5-6729-4584-b263-797d89678636 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:23.353258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30c63d5-6729-4584-b263-797d89678636 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:23.363458 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1430/2841] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:27:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:27:23.367505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03daa20-8678-435c-852a-e2b779fa3d79 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:23.377670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03daa20-8678-435c-852a-e2b779fa3d79 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:23.382593 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:23.383257 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:23.383532 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:23.383804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] PATH_INFO: `/v3/projects/5223392ed2dc4eff871b9e788e4d677e/users/8920a7e7625b46f59c3a89e2930f6535/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:23.398926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5223392ed2dc4eff871b9e788e4d677e, user_id=8920a7e7625b46f59c3a89e2930f6535, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:23.402094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:23.502542 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1412/2842] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:27:23 2022] PUT /identity/v3/projects/5223392ed2dc4eff871b9e788e4d677e/users/8920a7e7625b46f59c3a89e2930f6535/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:27:23.507832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52eacc1-cad3-4101-bda2-d11176fbcdb0 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:23.508101 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52eacc1-cad3-4101-bda2-d11176fbcdb0 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:23.508371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52eacc1-cad3-4101-bda2-d11176fbcdb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:23.528676 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-a52eacc1-cad3-4101-bda2-d11176fbcdb0 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:27:23.529910 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a52eacc1-cad3-4101-bda2-d11176fbcdb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:23.589616 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1431/2843] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:27:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:27:23.599339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:23.601083 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:23.632603 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:23.658659 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63bfa5d-679b-4d41-8626-9885c0607972 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:23.662364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c55ef5-b223-44d9-8ea7-de894ab44784 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:23.663036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c55ef5-b223-44d9-8ea7-de894ab44784 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:23.663319 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c55ef5-b223-44d9-8ea7-de894ab44784 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:23.663596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c55ef5-b223-44d9-8ea7-de894ab44784 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:23.672443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c55ef5-b223-44d9-8ea7-de894ab44784 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:23.673926 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c55ef5-b223-44d9-8ea7-de894ab44784 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:23.683675 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1413/2844] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:27:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:27:26.627890 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52eacc1-cad3-4101-bda2-d11176fbcdb0 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:26.629938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a52eacc1-cad3-4101-bda2-d11176fbcdb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:26.642761 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52eacc1-cad3-4101-bda2-d11176fbcdb0 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:26.647462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:26.648098 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:26.648544 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:26.648816 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] PATH_INFO: `/v3/users/3760151f95ce4c6c88f24b51ff3b6625` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:26.650480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3760151f95ce4c6c88f24b51ff3b6625)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:26.652393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:26.890333 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3760151f95ce4c6c88f24b51ff3b6625'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:26.891199 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3760151f95ce4c6c88f24b51ff3b6625'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:26.907774 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3760151f95ce4c6c88f24b51ff3b6625'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:27.033746 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3760151f95ce4c6c88f24b51ff3b6625'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:27.044688 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3760151f95ce4c6c88f24b51ff3b6625'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:27.047145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:27.060142 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055e5cb9-a1c3-42cc-8501-92e726d49c15 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:27.063109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055e5cb9-a1c3-42cc-8501-92e726d49c15 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:27.063109 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055e5cb9-a1c3-42cc-8501-92e726d49c15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:27.063298 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1432/2845] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:27:26 2022] DELETE /identity/v3/users/3760151f95ce4c6c88f24b51ff3b6625 => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:27:27.090938 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-055e5cb9-a1c3-42cc-8501-92e726d49c15 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:27:27.092541 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-055e5cb9-a1c3-42cc-8501-92e726d49c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:27.180408 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1414/2846] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:27:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 25 05:27:27.193344 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:27.194998 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:27.284156 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd7bf3-daa8-4eba-a00b-48a21602c965 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:27.288428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c61007-b4ac-4076-a5c0-5b2ccfd9ddaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:27.289161 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c61007-b4ac-4076-a5c0-5b2ccfd9ddaa service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:27.289547 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c61007-b4ac-4076-a5c0-5b2ccfd9ddaa service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:27.289960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c61007-b4ac-4076-a5c0-5b2ccfd9ddaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:27.300428 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c61007-b4ac-4076-a5c0-5b2ccfd9ddaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:27.302596 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c61007-b4ac-4076-a5c0-5b2ccfd9ddaa service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:27.314175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1433/2847] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:27:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:27:27.891683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055e5cb9-a1c3-42cc-8501-92e726d49c15 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:27.894057 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-055e5cb9-a1c3-42cc-8501-92e726d49c15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:27.937646 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055e5cb9-a1c3-42cc-8501-92e726d49c15 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:27.948046 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:27.949205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:27.950244 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:27.950773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] PATH_INFO: `/v3/projects/cac1cbc4cf3547288a7e5f6edea8e4c8` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:27.953323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cac1cbc4cf3547288a7e5f6edea8e4c8)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:27.959165 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:28.118884 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cac1cbc4cf3547288a7e5f6edea8e4c8', 'request_id': 'req-5b654963-8324-42b4-827f-7b6561371bb8'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:28.261754 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cac1cbc4cf3547288a7e5f6edea8e4c8', 'request_id': 'req-5b654963-8324-42b4-827f-7b6561371bb8'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:28.268522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cac1cbc4cf3547288a7e5f6edea8e4c8', 'request_id': 'req-5b654963-8324-42b4-827f-7b6561371bb8'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:28.306419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] The token cache is being invalidate because project cac1cbc4cf3547288a7e5f6edea8e4c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:27:28.306731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:28.323400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:27:28.323400 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1415/2848] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:27:27 2022] DELETE /identity/v3/projects/cac1cbc4cf3547288a7e5f6edea8e4c8 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:27:29.992467 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc125d2e-f238-44bf-a320-0dd0cea829f3 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:30.000145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc125d2e-f238-44bf-a320-0dd0cea829f3 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:30.000145 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc125d2e-f238-44bf-a320-0dd0cea829f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:30.030211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-cc125d2e-f238-44bf-a320-0dd0cea829f3 None None] MFA Rules not processed for user `8920a7e7625b46f59c3a89e2930f6535`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:27:30.031466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cc125d2e-f238-44bf-a320-0dd0cea829f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:30.108471 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1434/2849] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:27:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:27:30.113217 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:30.115081 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:30.179711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:30.193543 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b654963-8324-42b4-827f-7b6561371bb8 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:30.198723 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c554371-3220-426b-adcc-120617f3e885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:30.199429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c554371-3220-426b-adcc-120617f3e885 admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:30.199784 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c554371-3220-426b-adcc-120617f3e885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:30.200130 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c554371-3220-426b-adcc-120617f3e885 admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:30.200834 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c554371-3220-426b-adcc-120617f3e885 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:30.203040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c554371-3220-426b-adcc-120617f3e885 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:30.301964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1416/2850] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:27:30 2022] POST /identity/v3/projects => generated 377 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:27:30.306280 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc125d2e-f238-44bf-a320-0dd0cea829f3 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:30.317722 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc125d2e-f238-44bf-a320-0dd0cea829f3 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:30.388175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9849e907-0e15-46eb-abd4-d45e52169999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:30.389152 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9849e907-0e15-46eb-abd4-d45e52169999 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:30.389474 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9849e907-0e15-46eb-abd4-d45e52169999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:30.389917 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9849e907-0e15-46eb-abd4-d45e52169999 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:30.390549 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9849e907-0e15-46eb-abd4-d45e52169999 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:30.392505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9849e907-0e15-46eb-abd4-d45e52169999 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:30.468432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1435/2851] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:27:30 2022] POST /identity/v3/users => generated 295 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:27:30.472553 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c554371-3220-426b-adcc-120617f3e885 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:30.483254 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c554371-3220-426b-adcc-120617f3e885 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:30.487760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64db05c0-3c44-499a-8015-c9dc59840f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:30.488357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64db05c0-3c44-499a-8015-c9dc59840f2c admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:30.488635 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64db05c0-3c44-499a-8015-c9dc59840f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:30.490643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64db05c0-3c44-499a-8015-c9dc59840f2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:30.490643 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64db05c0-3c44-499a-8015-c9dc59840f2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:30.491227 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64db05c0-3c44-499a-8015-c9dc59840f2c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:30.500845 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1417/2852] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:27:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:27:30.504895 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9849e907-0e15-46eb-abd4-d45e52169999 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:30.515433 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9849e907-0e15-46eb-abd4-d45e52169999 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:30.520067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7028dc-2955-4e98-8f2b-9236870507eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:30.520711 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7028dc-2955-4e98-8f2b-9236870507eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:30.520989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7028dc-2955-4e98-8f2b-9236870507eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:30.521258 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7028dc-2955-4e98-8f2b-9236870507eb admin admin] PATH_INFO: `/v3/projects/5e8a8884da6245b58fb905708bf4ced5/users/32eb91e17151414d8c85da8707db0a2d/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:30.537445 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7028dc-2955-4e98-8f2b-9236870507eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e8a8884da6245b58fb905708bf4ced5, user_id=32eb91e17151414d8c85da8707db0a2d, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:30.542451 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7028dc-2955-4e98-8f2b-9236870507eb admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:30.544013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:30.645535 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1436/2853] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:27:30 2022] PUT /identity/v3/projects/5e8a8884da6245b58fb905708bf4ced5/users/32eb91e17151414d8c85da8707db0a2d/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:27:30.649506 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64db05c0-3c44-499a-8015-c9dc59840f2c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:30.659619 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64db05c0-3c44-499a-8015-c9dc59840f2c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:30.721380 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b50a71-4d4a-4129-abe8-098267c91fdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:30.724922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b50a71-4d4a-4129-abe8-098267c91fdb admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:30.724922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b50a71-4d4a-4129-abe8-098267c91fdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:30.724922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b50a71-4d4a-4129-abe8-098267c91fdb admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:30.724922 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b50a71-4d4a-4129-abe8-098267c91fdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:30.725722 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b50a71-4d4a-4129-abe8-098267c91fdb admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:30.736790 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1418/2854] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:27:30 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:27:30.741055 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7028dc-2955-4e98-8f2b-9236870507eb admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:30.752205 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7028dc-2955-4e98-8f2b-9236870507eb admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:30.757238 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:30.757913 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:30.758198 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:30.758472 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] PATH_INFO: `/v3/projects/5e8a8884da6245b58fb905708bf4ced5/users/32eb91e17151414d8c85da8707db0a2d/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:30.763432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e8a8884da6245b58fb905708bf4ced5, user_id=32eb91e17151414d8c85da8707db0a2d, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:30.763432 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:30.773100 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1437/2855] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:27:30 2022] PUT /identity/v3/projects/5e8a8884da6245b58fb905708bf4ced5/users/32eb91e17151414d8c85da8707db0a2d/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:27:36.432136 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c793e81a-06f4-4359-a4d7-2dba96b418f8 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:36.432655 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c793e81a-06f4-4359-a4d7-2dba96b418f8 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:36.432741 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c793e81a-06f4-4359-a4d7-2dba96b418f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:36.468753 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-c793e81a-06f4-4359-a4d7-2dba96b418f8 None None] MFA Rules not processed for user `32eb91e17151414d8c85da8707db0a2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:27:36.470024 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c793e81a-06f4-4359-a4d7-2dba96b418f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:36.551308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1419/2856] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:27:36 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:27:36.569319 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:36.571067 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:36.659954 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca75093-1c15-4a22-b89f-8eecaf6166ed admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:36.665915 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:36.666529 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:36.666806 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:36.667080 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:36.725244 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:36.727031 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:36.736585 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1438/2857] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:27:36 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:27:37.170733 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c793e81a-06f4-4359-a4d7-2dba96b418f8 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:37.172221 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-c793e81a-06f4-4359-a4d7-2dba96b418f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:37.232357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:37.236265 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c793e81a-06f4-4359-a4d7-2dba96b418f8 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:37.243623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ce0b85-32d5-4d32-a4b6-1fac9377b1a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:37.243623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce0b85-32d5-4d32-a4b6-1fac9377b1a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:37.243623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce0b85-32d5-4d32-a4b6-1fac9377b1a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:37.243623 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ce0b85-32d5-4d32-a4b6-1fac9377b1a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:37.244155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-27ce0b85-32d5-4d32-a4b6-1fac9377b1a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:37.306522 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ce0b85-32d5-4d32-a4b6-1fac9377b1a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:37.307966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ce0b85-32d5-4d32-a4b6-1fac9377b1a3 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:37.319744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1420/2858] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:27:37 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:27:43.206146 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:27:43.207557 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:43.288439 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6681273-d0de-469a-9d41-2d95d78c7643 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:27:43.293466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:27:43.294103 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:27:43.294381 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:27:43.294652 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:27:43.296376 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:27:43.308528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:27:43.310213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:27:43.321405 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1439/2859] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:27:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:27:47.570664 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=28 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:27:53.317076 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:28:03.621964 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fbc00-d3ca-4eca-a019-379af1bf8823 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:28:03.622581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fbc00-d3ca-4eca-a019-379af1bf8823 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:28:03.622581 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143fbc00-d3ca-4eca-a019-379af1bf8823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:28:03.641656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-143fbc00-d3ca-4eca-a019-379af1bf8823 None None] MFA Rules not processed for user `8920a7e7625b46f59c3a89e2930f6535`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:28:03.642854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-143fbc00-d3ca-4eca-a019-379af1bf8823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:28:03.650242 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1421/2860] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:28:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:28:13.636389 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:28:24.651525 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:28:24.652914 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:28:24.664028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885ca929-7486-43be-9dea-e685d085b759 service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:28:24.668083 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be8dcbb-949c-4e29-b023-6b78ae598b19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:28:24.668666 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8dcbb-949c-4e29-b023-6b78ae598b19 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:28:24.668943 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8dcbb-949c-4e29-b023-6b78ae598b19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:28:24.669211 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8dcbb-949c-4e29-b023-6b78ae598b19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:28:24.677950 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be8dcbb-949c-4e29-b023-6b78ae598b19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:28:24.679403 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be8dcbb-949c-4e29-b023-6b78ae598b19 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:28:24.689503 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1440/2861] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:28:24 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:28:26.795216 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143fbc00-d3ca-4eca-a019-379af1bf8823 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:28:26.807931 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143fbc00-d3ca-4eca-a019-379af1bf8823 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:28:26.813672 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:28:26.814292 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:28:26.814573 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:28:26.814873 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:28:26.816616 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:28:26.830023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:28:26.831860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:28:26.843480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1422/2862] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:28:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:28:34.685315 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:28:36.838870 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:28:55.505801 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be8dcbb-949c-4e29-b023-6b78ae598b19 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:28:55.515217 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be8dcbb-949c-4e29-b023-6b78ae598b19 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:28:55.518779 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:28:55.519452 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:28:55.519731 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:28:55.520011 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:28:55.521733 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:28:55.535020 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:28:55.536424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:28:55.547516 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1441/2863] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:28:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:28:56.739342 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:28:56.748757 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e7559c-a4a2-42a1-9bf2-84c697d28d2b service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:28:56.752313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f1d137-a2d2-44f0-8a34-bd5593144d50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:28:56.752963 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1d137-a2d2-44f0-8a34-bd5593144d50 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:28:56.753684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1d137-a2d2-44f0-8a34-bd5593144d50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:28:56.753684 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1d137-a2d2-44f0-8a34-bd5593144d50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:28:56.755272 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-97f1d137-a2d2-44f0-8a34-bd5593144d50 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:28:56.769711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f1d137-a2d2-44f0-8a34-bd5593144d50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:28:56.771143 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f1d137-a2d2-44f0-8a34-bd5593144d50 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:28:56.782897 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1423/2864] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:28:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:29:05.543147 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:29:06.777773 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:30:13.360187 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:13.361707 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:13.410365 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2940f377-7c3a-4df5-833a-7ca48da37660 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:13.417780 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:13.418371 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:13.418695 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:13.418939 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:13.420795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:13.433206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:13.434802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:13.446377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1442/2865] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:30:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:30:16.189878 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54ad02a-88f7-4ded-8e77-b6827b750dc9 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:16.190327 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54ad02a-88f7-4ded-8e77-b6827b750dc9 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:16.190434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54ad02a-88f7-4ded-8e77-b6827b750dc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:16.210038 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-f54ad02a-88f7-4ded-8e77-b6827b750dc9 None None] MFA Rules not processed for user `b7ba48fe350b4644a9f5edd0729a0536`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:30:16.211236 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f54ad02a-88f7-4ded-8e77-b6827b750dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:16.267199 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1424/2866] 10.222.0.16 () {58 vars in 967 bytes} [Thu Aug 25 05:30:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:30:16.306375 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:16.308188 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:16.376971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db9470d-1aa8-4c76-b502-3035f25965e4 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:16.382775 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c77473684c58457aa53f73a9c30319d0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:16.383380 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:16.383654 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:16.383932 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:16.392038 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:16.393625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:16.401423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1443/2867] 10.222.0.16 () {60 vars in 1384 bytes} [Thu Aug 25 05:30:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 25 05:30:20.624011 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54ad02a-88f7-4ded-8e77-b6827b750dc9 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:20.636206 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54ad02a-88f7-4ded-8e77-b6827b750dc9 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:20.641167 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:20.641815 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:20.642097 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:20.642370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:20.644098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:20.655755 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:20.657341 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:20.668308 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1425/2868] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:30:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:30:24.053399 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:24.054860 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:24.154462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855a9d29-1745-4777-96e1-db25b212c4cb service swift] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:24.159318 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:24.159976 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:24.160354 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:24.160576 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] PATH_INFO: `/v3/users/fb70879694d443df9b30ff172cfa39b4` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:24.161727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb70879694d443df9b30ff172cfa39b4)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:24.171795 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:24.181352 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:30:24.380476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb70879694d443df9b30ff172cfa39b4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.381127 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb70879694d443df9b30ff172cfa39b4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.393265 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb70879694d443df9b30ff172cfa39b4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.472355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb70879694d443df9b30ff172cfa39b4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.479989 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb70879694d443df9b30ff172cfa39b4'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.485965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1444/2869] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:30:24 2022] DELETE /identity/v3/users/fb70879694d443df9b30ff172cfa39b4 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:30:24.731887 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:24.743114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf8c799-892f-4a03-97ae-ef7daa8e0c98 service placement] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:24.800599 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:24.801829 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:24.802106 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:24.802374 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] PATH_INFO: `/v3/projects/b0f35b076ec94efda11070e02d3a5acd` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:24.803478 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0f35b076ec94efda11070e02d3a5acd)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:24.805360 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:24.912670 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0f35b076ec94efda11070e02d3a5acd', 'request_id': 'req-dc763ba0-1bef-41d5-b580-d15ace62ac56'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.980854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0f35b076ec94efda11070e02d3a5acd', 'request_id': 'req-dc763ba0-1bef-41d5-b580-d15ace62ac56'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.986860 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0f35b076ec94efda11070e02d3a5acd', 'request_id': 'req-dc763ba0-1bef-41d5-b580-d15ace62ac56'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.998656 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] The token cache is being invalidate because project b0f35b076ec94efda11070e02d3a5acd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:30:24.998947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:24.999609 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.001077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1426/2870] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:30:24 2022] DELETE /identity/v3/projects/b0f35b076ec94efda11070e02d3a5acd => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:30:25.005194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:25.006614 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:25.019355 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cbc275-e21f-422d-a784-456357881835 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:25.075180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:25.075798 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:25.076084 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:25.076361 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] PATH_INFO: `/v3/users/4f4a3d3b68dc481c8e4a099bde99d686` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:25.077492 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f4a3d3b68dc481c8e4a099bde99d686)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:25.079443 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:25.271767 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f4a3d3b68dc481c8e4a099bde99d686'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.272367 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f4a3d3b68dc481c8e4a099bde99d686'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.284623 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f4a3d3b68dc481c8e4a099bde99d686'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.366373 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f4a3d3b68dc481c8e4a099bde99d686'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.373778 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-0a3ae8fd-65a0-4af3-b7a8-700fe94b3c9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f4a3d3b68dc481c8e4a099bde99d686'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.379684 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1445/2871] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:30:25 2022] DELETE /identity/v3/users/4f4a3d3b68dc481c8e4a099bde99d686 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:30:25.642590 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:25.647304 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:25.657538 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc763ba0-1bef-41d5-b580-d15ace62ac56 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:25.683238 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:30:25.721701 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:25.722333 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:25.722615 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:25.722880 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] PATH_INFO: `/v3/projects/52469a83e4f74020b1208ed6ab573320` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:25.724053 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] RBAC: Authorizing `identity:delete_project(project_id=52469a83e4f74020b1208ed6ab573320)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:25.726345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:25.827075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52469a83e4f74020b1208ed6ab573320', 'request_id': 'req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.909637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52469a83e4f74020b1208ed6ab573320', 'request_id': 'req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.915443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52469a83e4f74020b1208ed6ab573320', 'request_id': 'req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.929230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] The token cache is being invalidate because project 52469a83e4f74020b1208ed6ab573320 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:30:25.929524 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.930245 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-ea2fec2e-8eae-4f80-a2af-2b7788409ebf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:30:25.931724 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1427/2872] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:30:25 2022] DELETE /identity/v3/projects/52469a83e4f74020b1208ed6ab573320 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:30:25.936852 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1df6062-7272-4fe2-b4ab-3b73e0e050fb None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:25.937213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1df6062-7272-4fe2-b4ab-3b73e0e050fb None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:25.937493 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1df6062-7272-4fe2-b4ab-3b73e0e050fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:25.962993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:30:25.965709 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-e1df6062-7272-4fe2-b4ab-3b73e0e050fb None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:30:25.966894 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-e1df6062-7272-4fe2-b4ab-3b73e0e050fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:26.024938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1446/2873] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:30:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:30:26.032695 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a839c3c-e466-4c99-a8d7-0d2050510f06 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.032986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a839c3c-e466-4c99-a8d7-0d2050510f06 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.033263 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a839c3c-e466-4c99-a8d7-0d2050510f06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.052955 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-8a839c3c-e466-4c99-a8d7-0d2050510f06 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:30:26.054188 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-8a839c3c-e466-4c99-a8d7-0d2050510f06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:26.062056 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1428/2874] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:30:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:30:26.066193 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1df6062-7272-4fe2-b4ab-3b73e0e050fb None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:26.076229 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1df6062-7272-4fe2-b4ab-3b73e0e050fb None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:26.080701 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:26.081287 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.081561 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.081934 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] PATH_INFO: `/v3/domains` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.082571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81762) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 25 05:30:26.082870 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:26.084444 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:26.095969 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1447/2875] 10.222.0.16 () {58 vars in 1153 bytes} [Thu Aug 25 05:30:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 25 05:30:26.100416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a839c3c-e466-4c99-a8d7-0d2050510f06 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:26.109508 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a839c3c-e466-4c99-a8d7-0d2050510f06 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:26.114705 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b21ff5-37d0-4cb9-b59a-89a599d63d1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:26.115296 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b21ff5-37d0-4cb9-b59a-89a599d63d1c admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.115575 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b21ff5-37d0-4cb9-b59a-89a599d63d1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.115847 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b21ff5-37d0-4cb9-b59a-89a599d63d1c admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.116464 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b21ff5-37d0-4cb9-b59a-89a599d63d1c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:26.118403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b21ff5-37d0-4cb9-b59a-89a599d63d1c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:26.223226 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1429/2876] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:30:26 2022] POST /identity/v3/projects => generated 373 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:30:26.227670 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:26.237862 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69306b78-e09a-4b2d-bec9-0ace7f0f5bdf admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:26.295309 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7600478b-37bf-4fc8-9fd0-f4d71a502d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:26.295942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7600478b-37bf-4fc8-9fd0-f4d71a502d3d admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.296228 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7600478b-37bf-4fc8-9fd0-f4d71a502d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.296494 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7600478b-37bf-4fc8-9fd0-f4d71a502d3d admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.297117 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7600478b-37bf-4fc8-9fd0-f4d71a502d3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:26.299058 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7600478b-37bf-4fc8-9fd0-f4d71a502d3d admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:26.419168 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1448/2877] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:30:26 2022] POST /identity/v3/users => generated 287 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 25 05:30:26.423194 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b21ff5-37d0-4cb9-b59a-89a599d63d1c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:26.434344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b21ff5-37d0-4cb9-b59a-89a599d63d1c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:26.438965 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d645a40-13af-4b18-8414-10eaacd1b22c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:26.439563 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d645a40-13af-4b18-8414-10eaacd1b22c admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.439843 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d645a40-13af-4b18-8414-10eaacd1b22c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.440114 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d645a40-13af-4b18-8414-10eaacd1b22c admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.440777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d645a40-13af-4b18-8414-10eaacd1b22c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:26.442416 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d645a40-13af-4b18-8414-10eaacd1b22c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:26.452668 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1430/2878] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:30:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:30:26.456710 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7600478b-37bf-4fc8-9fd0-f4d71a502d3d admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:26.466571 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7600478b-37bf-4fc8-9fd0-f4d71a502d3d admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:26.471727 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41743da-20d5-4892-a6af-5cd147d2a974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:26.472332 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41743da-20d5-4892-a6af-5cd147d2a974 admin admin] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.473197 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41743da-20d5-4892-a6af-5cd147d2a974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.473422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41743da-20d5-4892-a6af-5cd147d2a974 admin admin] PATH_INFO: `/v3/roles` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.474123 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41743da-20d5-4892-a6af-5cd147d2a974 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:26.476093 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41743da-20d5-4892-a6af-5cd147d2a974 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:26.487045 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1449/2879] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:30:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:30:26.491879 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d645a40-13af-4b18-8414-10eaacd1b22c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:26.501936 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d645a40-13af-4b18-8414-10eaacd1b22c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:26.506502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:26.507153 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.507431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.507704 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] PATH_INFO: `/v3/projects/52aad667fe8d4b6582a7e4340ecfe33f/users/8a77b0db996b4bd58f2a92344a26fe78/roles/0940d9778fb041fbaf7c4f4317feb2d5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.522316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52aad667fe8d4b6582a7e4340ecfe33f, user_id=8a77b0db996b4bd58f2a92344a26fe78, role_id=0940d9778fb041fbaf7c4f4317feb2d5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:26.524933 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:26.589313 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1431/2880] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:30:26 2022] PUT /identity/v3/projects/52aad667fe8d4b6582a7e4340ecfe33f/users/8a77b0db996b4bd58f2a92344a26fe78/roles/0940d9778fb041fbaf7c4f4317feb2d5 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:30:26.594668 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf9ca34-2184-40df-b4d3-e0dd91cf1f65 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.594951 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf9ca34-2184-40df-b4d3-e0dd91cf1f65 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.595220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf9ca34-2184-40df-b4d3-e0dd91cf1f65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.614960 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-eaf9ca34-2184-40df-b4d3-e0dd91cf1f65 None None] MFA Rules not processed for user `c2010f49b97347fb8aff29794f27d93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:30:26.616150 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-eaf9ca34-2184-40df-b4d3-e0dd91cf1f65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:26.674182 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1450/2881] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:30:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:30:26.688000 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:26.689785 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:26.741904 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:30:26.745890 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998628e1-82da-402b-8773-f26bc4640b49 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:26.749697 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff3f1d4-9773-4507-b344-0fa22febf749 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:26.754730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3f1d4-9773-4507-b344-0fa22febf749 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:26.754730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3f1d4-9773-4507-b344-0fa22febf749 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:26.754730 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff3f1d4-9773-4507-b344-0fa22febf749 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:26.759744 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff3f1d4-9773-4507-b344-0fa22febf749 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:26.761212 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff3f1d4-9773-4507-b344-0fa22febf749 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:26.770994 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1432/2882] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:30:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:30:32.747275 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8e5e99-43d7-47a7-a243-2e7bb4c78493 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:32.747803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8e5e99-43d7-47a7-a243-2e7bb4c78493 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:32.747803 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8e5e99-43d7-47a7-a243-2e7bb4c78493 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:32.778291 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:30:32.785869 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-7e8e5e99-43d7-47a7-a243-2e7bb4c78493 None None] MFA Rules not processed for user `8a77b0db996b4bd58f2a92344a26fe78`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:30:32.785869 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-7e8e5e99-43d7-47a7-a243-2e7bb4c78493 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:32.865028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1451/2883] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:30:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:30:32.869663 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff3f1d4-9773-4507-b344-0fa22febf749 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:32.893537 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff3f1d4-9773-4507-b344-0fa22febf749 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:32.900239 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b2163-c712-477d-8f78-d4691f033f6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:32.901013 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b2163-c712-477d-8f78-d4691f033f6f admin admin] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:32.901358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b2163-c712-477d-8f78-d4691f033f6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:32.901760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736b2163-c712-477d-8f78-d4691f033f6f admin admin] PATH_INFO: `/v3/projects` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:32.906377 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736b2163-c712-477d-8f78-d4691f033f6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:32.912094 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736b2163-c712-477d-8f78-d4691f033f6f admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:33.060596 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1433/2884] 10.222.0.16 () {60 vars in 1153 bytes} [Thu Aug 25 05:30:32 2022] POST /identity/v3/projects => generated 371 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:30:33.064426 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8e5e99-43d7-47a7-a243-2e7bb4c78493 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:33.076213 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8e5e99-43d7-47a7-a243-2e7bb4c78493 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:33.137551 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e398910-0ccc-4a38-8cb9-0512ece8df09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:33.138282 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e398910-0ccc-4a38-8cb9-0512ece8df09 admin admin] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:33.138562 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e398910-0ccc-4a38-8cb9-0512ece8df09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:33.138832 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e398910-0ccc-4a38-8cb9-0512ece8df09 admin admin] PATH_INFO: `/v3/users` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:33.139476 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e398910-0ccc-4a38-8cb9-0512ece8df09 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:33.141422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e398910-0ccc-4a38-8cb9-0512ece8df09 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:33.256412 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1452/2885] 10.222.0.16 () {60 vars in 1144 bytes} [Thu Aug 25 05:30:33 2022] POST /identity/v3/users => generated 292 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 25 05:30:33.264732 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b2163-c712-477d-8f78-d4691f033f6f admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:33.271357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736b2163-c712-477d-8f78-d4691f033f6f admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:33.275971 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73538044-3615-469b-b6d7-a85ac82711db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:33.276585 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73538044-3615-469b-b6d7-a85ac82711db admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:33.276867 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73538044-3615-469b-b6d7-a85ac82711db admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:33.277170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73538044-3615-469b-b6d7-a85ac82711db admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:33.277910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73538044-3615-469b-b6d7-a85ac82711db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:33.279505 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73538044-3615-469b-b6d7-a85ac82711db admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:33.293622 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1434/2886] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:30:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 25 05:30:33.296294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e398910-0ccc-4a38-8cb9-0512ece8df09 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:33.307203 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e398910-0ccc-4a38-8cb9-0512ece8df09 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:33.312729 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc451638-163f-4c5c-bade-d624ed3c764b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:33.313336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc451638-163f-4c5c-bade-d624ed3c764b admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:33.313648 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc451638-163f-4c5c-bade-d624ed3c764b admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:33.313956 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc451638-163f-4c5c-bade-d624ed3c764b admin admin] PATH_INFO: `/v3/projects/a1e042d8df5e4494a03603ae00efd15b/users/fb4c7c20d4564097acec780f97479d3d/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:33.329625 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc451638-163f-4c5c-bade-d624ed3c764b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1e042d8df5e4494a03603ae00efd15b, user_id=fb4c7c20d4564097acec780f97479d3d, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:33.332743 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc451638-163f-4c5c-bade-d624ed3c764b admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:33.436259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1453/2887] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:30:33 2022] PUT /identity/v3/projects/a1e042d8df5e4494a03603ae00efd15b/users/fb4c7c20d4564097acec780f97479d3d/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:30:33.440442 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73538044-3615-469b-b6d7-a85ac82711db admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:33.462509 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73538044-3615-469b-b6d7-a85ac82711db admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:33.518104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf77f4a9-8d4c-43fa-8c56-f407c945be33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:33.518751 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77f4a9-8d4c-43fa-8c56-f407c945be33 admin admin] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:33.519093 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77f4a9-8d4c-43fa-8c56-f407c945be33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:33.519782 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77f4a9-8d4c-43fa-8c56-f407c945be33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:33.520436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf77f4a9-8d4c-43fa-8c56-f407c945be33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:33.522140 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf77f4a9-8d4c-43fa-8c56-f407c945be33 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:33.536358 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1435/2888] 10.222.0.16 () {58 vars in 1122 bytes} [Thu Aug 25 05:30:33 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 25 05:30:33.540290 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc451638-163f-4c5c-bade-d624ed3c764b admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:33.552466 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc451638-163f-4c5c-bade-d624ed3c764b admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:33.557488 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:33.558377 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:33.560802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:33.560802 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] PATH_INFO: `/v3/projects/a1e042d8df5e4494a03603ae00efd15b/users/fb4c7c20d4564097acec780f97479d3d/roles/301b10bcbece4095821ce8c90c3e3332` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:33.561001 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1e042d8df5e4494a03603ae00efd15b, user_id=fb4c7c20d4564097acec780f97479d3d, role_id=301b10bcbece4095821ce8c90c3e3332)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:33.563593 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:33.574984 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1454/2889] 10.222.0.16 () {60 vars in 1483 bytes} [Thu Aug 25 05:30:33 2022] PUT /identity/v3/projects/a1e042d8df5e4494a03603ae00efd15b/users/fb4c7c20d4564097acec780f97479d3d/roles/301b10bcbece4095821ce8c90c3e3332 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:30:39.662706 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515a211-59da-4f36-aa58-d71d51e7c46c None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:39.663225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515a211-59da-4f36-aa58-d71d51e7c46c None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:39.663225 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2515a211-59da-4f36-aa58-d71d51e7c46c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:39.695359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-2515a211-59da-4f36-aa58-d71d51e7c46c None None] MFA Rules not processed for user `fb4c7c20d4564097acec780f97479d3d`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:30:39.696740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-2515a211-59da-4f36-aa58-d71d51e7c46c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:39.780558 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1436/2890] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:30:39 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:30:39.796680 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:39.799646 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:39.866402 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:30:39.889462 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f009ab82-d858-49fb-9615-7485acc9c2ae admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:39.898615 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1db156-90ba-465f-98d7-0a2a116d5d9b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:39.899239 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1db156-90ba-465f-98d7-0a2a116d5d9b service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:39.899528 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1db156-90ba-465f-98d7-0a2a116d5d9b service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:39.899777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1db156-90ba-465f-98d7-0a2a116d5d9b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:39.968111 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1db156-90ba-465f-98d7-0a2a116d5d9b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:39.972350 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1db156-90ba-465f-98d7-0a2a116d5d9b service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:39.982114 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1455/2891] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:30:39 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:30:42.563189 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4816e0f-2c17-4420-858a-7edd30b9f953 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:42.564170 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4816e0f-2c17-4420-858a-7edd30b9f953 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:42.564453 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4816e0f-2c17-4420-858a-7edd30b9f953 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:42.584826 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-e4816e0f-2c17-4420-858a-7edd30b9f953 None None] MFA Rules not processed for user `8a77b0db996b4bd58f2a92344a26fe78`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:30:42.586293 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e4816e0f-2c17-4420-858a-7edd30b9f953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:42.593230 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1437/2892] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:30:42 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:30:49.634447 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aef9f54-a943-4cad-86b2-242937dc6400 None None] REQUEST_METHOD: `POST` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:49.634965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aef9f54-a943-4cad-86b2-242937dc6400 None None] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:49.635074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aef9f54-a943-4cad-86b2-242937dc6400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:49.654422 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.auth.core [None req-4aef9f54-a943-4cad-86b2-242937dc6400 None None] MFA Rules not processed for user `8920a7e7625b46f59c3a89e2930f6535`. Rule list: `[]` (Enabled: `True`). {{(pid=81762) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:30:49.655611 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4aef9f54-a943-4cad-86b2-242937dc6400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:49.711590 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1456/2893] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:30:49 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:30:51.011438 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4816e0f-2c17-4420-858a-7edd30b9f953 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:51.012901 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-e4816e0f-2c17-4420-858a-7edd30b9f953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:51.052264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:30:51.086969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4816e0f-2c17-4420-858a-7edd30b9f953 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:51.093551 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:51.094203 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:51.094480 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:51.094746 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:51.096502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:51.108495 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:51.110098 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:51.119682 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1438/2894] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:30:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:30:54.027220 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aef9f54-a943-4cad-86b2-242937dc6400 None None] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:30:54.028837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-4aef9f54-a943-4cad-86b2-242937dc6400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:54.087508 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aef9f54-a943-4cad-86b2-242937dc6400 None None] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:30:54.091175 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:30:54.091777 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:30:54.092056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:30:54.092328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:30:54.094063 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:30:54.104026 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:30:54.105553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:30:54.113475 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1457/2895] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:30:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:31:01.123023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=17 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:31:04.109941 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=82 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:31:07.901803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:31:07.915917 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9b1a39-e70f-4dfd-a1e9-47dc6d9d0fe4 service placement] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:31:07.923443 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:31:07.924190 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:31:07.924323 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:31:07.924760 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:31:07.926549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:31:07.939370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:31:07.941028 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:31:07.952027 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1439/2896] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:31:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:31:17.947424 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:31:24.451966 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:31:24.460642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd29a99a-c2ae-4422-9226-eaf7eb2d4e12 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:31:24.464219 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:31:24.464812 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:31:24.465091 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:31:24.465364 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:31:24.474087 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:31:24.475756 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:31:24.485294 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1458/2897] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:31:24 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:31:34.481252 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=6 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:32:44.287777 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:32:44.289168 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:32:44.340253 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f199296f-b70e-44e5-bdc0-904b488d0439 service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:32:44.345316 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15f8166-7218-4217-8f63-060e8b769993 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:32:44.345997 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15f8166-7218-4217-8f63-060e8b769993 service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:32:44.364211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15f8166-7218-4217-8f63-060e8b769993 service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:32:44.364211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15f8166-7218-4217-8f63-060e8b769993 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:32:44.364211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-d15f8166-7218-4217-8f63-060e8b769993 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:32:44.364211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15f8166-7218-4217-8f63-060e8b769993 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:32:44.364211 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15f8166-7218-4217-8f63-060e8b769993 service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:32:44.371714 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1440/2898] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:32:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:32:54.367549 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=22 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:33:02.100397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:33:02.101825 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:02.166691 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791b104-7b46-45e5-bf5a-17ed52a1861b service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:33:02.170565 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ec312d29dfa846cbace53e8444b8a527', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:33:02.174938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:02.174938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:02.174938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:02.174938 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:02.185018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:33:02.186539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:33:02.195270 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1459/2899] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:33:02 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:33:12.191191 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=29 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:33:13.882854 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30835492-130a-4aa7-b338-de00b2030906 None None] REQUEST_METHOD: `POST` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:13.883378 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30835492-130a-4aa7-b338-de00b2030906 None None] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:13.883491 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30835492-130a-4aa7-b338-de00b2030906 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:13.903182 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.auth.core [None req-30835492-130a-4aa7-b338-de00b2030906 None None] MFA Rules not processed for user `8920a7e7625b46f59c3a89e2930f6535`. Rule list: `[]` (Enabled: `True`). {{(pid=81763) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 25 05:33:13.904364 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-30835492-130a-4aa7-b338-de00b2030906 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:13.911467 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1441/2900] 10.222.0.16 () {58 vars in 958 bytes} [Thu Aug 25 05:33:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 25 05:33:14.235160 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:33:14.254999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96dcc41-ed88-424c-beec-f1f1747fdd45 service cinder] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:33:14.258662 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:33:14.259259 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:14.259539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:14.259813 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:14.261509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:14.283506 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:33:14.284929 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:33:14.293782 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1460/2901] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:33:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:33:23.896979 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=3 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:33:24.289905 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=15 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:33:38.236557 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30835492-130a-4aa7-b338-de00b2030906 None None] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:33:38.246359 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30835492-130a-4aa7-b338-de00b2030906 None None] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:33:38.249851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:33:38.250440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:38.250711 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:38.250973 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:38.252626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:38.314902 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:33:38.316331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:33:38.326429 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1442/2902] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:33:38 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:33:48.321534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=30 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:33:56.780421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:33:56.789013 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb79625-5153-4745-9329-5eb70e8a3cfe service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:33:56.793074 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:33:56.793689 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:56.793965 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:56.794231 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:56.795911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:56.830056 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:33:56.831567 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:33:56.851605 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1461/2903] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:33:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:33:57.036694 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:33:57.046231 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b5d514-ed19-4902-a120-adff06b78206 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:33:57.050039 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:33:57.050647 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:57.051075 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:57.051357 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:57.053339 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:57.064528 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:33:57.066026 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:33:57.074345 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1443/2904] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:33:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:33:58.506075 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:33:58.514855 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7da19bf-f593-4619-a651-d2b79024b4a4 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:33:58.518674 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:33:58.519348 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:58.519633 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:58.519911 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:58.521738 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:58.532788 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:33:58.534297 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:33:58.543336 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1462/2905] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:33:58 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:33:58.625721 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:33:58.634403 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f125692a-286b-4f36-b4e9-55e799e9af77 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:33:58.638633 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:33:58.639277 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:58.639502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:58.639771 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:58.641468 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:58.655195 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:33:58.660895 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:33:58.668144 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1444/2906] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:33:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:33:59.618181 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:33:59.627393 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597b0360-c71c-41a6-b7bc-f77fd9bdf140 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:33:59.630971 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:33:59.631553 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:33:59.631828 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:33:59.632096 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:33:59.633837 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:33:59.647394 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:33:59.648791 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:33:59.659505 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1463/2907] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:33:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:34:08.662638 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=12 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:34:09.654703 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=30 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:35:30.595637 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:30.614548 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a96611d-e986-4192-8d34-cbfbd41a7a7a service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:30.614970 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:30.615481 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:30.615959 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:30.616947 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:30.618851 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:30.632307 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:30.633939 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:30.647804 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1445/2908] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:35:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:35:40.642022 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=6 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:35:44.850343 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:44.859757 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3979ceeb-2452-4a70-abb0-5d5d6192e65a service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:44.863400 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:44.863993 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:44.864269 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:44.864539 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:44.866292 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:44.928289 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:44.929742 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:44.939659 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1464/2909] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:35:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:35:48.381626 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:48.393871 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f594c7-1eaf-4259-b33b-42896ca157eb service nova] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:48.399149 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:48.399875 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:48.400222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:48.400564 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:48.403431 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:48.415801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:48.417520 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:48.428617 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1446/2910] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:35:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:35:51.872530 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:51.884357 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec91023-60c2-428c-aebf-600ad505ce1d service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:51.890369 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:51.891518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:51.891518 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:51.891995 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:51.893783 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:51.905378 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:51.906999 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:51.917512 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1465/2911] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:35:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:35:51.995019 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:51.996497 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:52.009681 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77f7152-eea7-465f-81f1-d999655d4ccf service placement] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:52.015179 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:52.015801 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:52.016078 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:52.016344 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] PATH_INFO: `/v3/users/8920a7e7625b46f59c3a89e2930f6535` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:52.017436 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8920a7e7625b46f59c3a89e2930f6535)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:52.019371 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:52.197061 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8920a7e7625b46f59c3a89e2930f6535'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.208740 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8920a7e7625b46f59c3a89e2930f6535'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.209395 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8920a7e7625b46f59c3a89e2930f6535'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.281023 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8920a7e7625b46f59c3a89e2930f6535'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.288502 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8920a7e7625b46f59c3a89e2930f6535'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.295104 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1447/2912] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:35:51 2022] DELETE /identity/v3/users/8920a7e7625b46f59c3a89e2930f6535 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:35:52.511480 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:52.521094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23e8ad8-b3b8-428d-8a3e-53158ff9922a service placement] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:52.562765 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:52.565155 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:52.571844 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:52.573047 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:52.573536 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:52.573974 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:52.575906 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:52.637977 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:52.639423 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:52.640119 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d54a51-ef11-4c7a-b6aa-604d1951caa5 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:52.644876 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:52.645621 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:52.645877 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:52.646148 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] PATH_INFO: `/v3/projects/5223392ed2dc4eff871b9e788e4d677e` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:52.647264 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5223392ed2dc4eff871b9e788e4d677e)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:52.648627 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1466/2913] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:35:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:35:52.650346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:52.756534 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5223392ed2dc4eff871b9e788e4d677e', 'request_id': 'req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.829440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5223392ed2dc4eff871b9e788e4d677e', 'request_id': 'req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.833841 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:35:52.836077 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5223392ed2dc4eff871b9e788e4d677e', 'request_id': 'req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.847966 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] The token cache is being invalidate because project 5223392ed2dc4eff871b9e788e4d677e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:35:52.848258 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.848909 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:52.850805 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1448/2914] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:35:52 2022] DELETE /identity/v3/projects/5223392ed2dc4eff871b9e788e4d677e => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 25 05:35:52.854927 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:52.865771 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ee5f83-f78f-4ab2-9dfc-9adeaf6d77ed service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:52.898284 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=100 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:35:52.925202 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:52.926167 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:52.926449 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:52.927088 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] PATH_INFO: `/v3/users/32eb91e17151414d8c85da8707db0a2d` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:52.927985 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=32eb91e17151414d8c85da8707db0a2d)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:52.929996 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:53.121435 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32eb91e17151414d8c85da8707db0a2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.122105 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32eb91e17151414d8c85da8707db0a2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.134004 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32eb91e17151414d8c85da8707db0a2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.211180 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32eb91e17151414d8c85da8707db0a2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.220094 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32eb91e17151414d8c85da8707db0a2d'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.232052 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1467/2915] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:35:52 2022] DELETE /identity/v3/users/32eb91e17151414d8c85da8707db0a2d => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:35:53.476639 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:53.478340 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:53.494161 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9145b8d9-5fba-4d40-ab4e-b9bc83ea958d admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:53.559803 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:53.560916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:53.560916 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:53.561154 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] PATH_INFO: `/v3/projects/5e8a8884da6245b58fb905708bf4ced5` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:53.562205 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5e8a8884da6245b58fb905708bf4ced5)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:53.571425 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:53.602958 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:53.604882 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:53.696969 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5e8a8884da6245b58fb905708bf4ced5', 'request_id': 'req-3f6c6a95-5366-4eaa-a71e-a977510936a6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.700339 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2656aad3-8af4-4ba8-b28d-680886249aa8 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:53.755347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:35:53.757018 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5e8a8884da6245b58fb905708bf4ced5', 'request_id': 'req-3f6c6a95-5366-4eaa-a71e-a977510936a6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.764679 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5e8a8884da6245b58fb905708bf4ced5', 'request_id': 'req-3f6c6a95-5366-4eaa-a71e-a977510936a6'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.776347 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb5ac6ef2f5048d3aed0d7a91abc4cac', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader', 'service', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:53.776979 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:53.777324 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:53.777555 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:53.779587 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:53.784881 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] The token cache is being invalidate because project 5e8a8884da6245b58fb905708bf4ced5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81763) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:35:53.785181 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.785856 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:35:53.787370 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1449/2916] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:35:53 2022] DELETE /identity/v3/projects/5e8a8884da6245b58fb905708bf4ced5 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:35:53.796787 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:53.799706 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:53.828638 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1468/2917] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:35:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 25 05:35:53.979423 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:53.985099 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:54.098824 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6c6a95-5366-4eaa-a71e-a977510936a6 admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:54.103499 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:54.107383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:54.107383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:54.107383 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:54.107683 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:54.118294 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:54.120484 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:54.131346 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1450/2918] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:35:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:35:59.009672 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:35:59.011149 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:59.022448 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc68981-3598-402a-914e-96e44afd27e0 service nova] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:35:59.026046 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9220ba3637a542a382ad1d42ff1ff459', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:35:59.026632 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] REQUEST_METHOD: `GET` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:35:59.026907 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:35:59.027177 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:35:59.028790 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:35:59.091328 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:35:59.092750 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:35:59.102509 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1469/2919] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:35:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:36:04.125787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:DELETE=8,keystone:SELECT=81,keystone:INSERT=1 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:36:06.115471 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:36:06.116986 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:36:06.131477 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81282c37-3e2a-4379-b2f3-7cef6c4bd590 service neutron] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:36:06.136379 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:36:06.137036 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:36:06.137315 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:36:06.137612 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] PATH_INFO: `/v3/users/8a77b0db996b4bd58f2a92344a26fe78` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:36:06.138750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a77b0db996b4bd58f2a92344a26fe78)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:36:06.140686 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:36:06.299910 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a77b0db996b4bd58f2a92344a26fe78'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.312222 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a77b0db996b4bd58f2a92344a26fe78'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.312892 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a77b0db996b4bd58f2a92344a26fe78'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.371787 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a77b0db996b4bd58f2a92344a26fe78'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.379903 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a77b0db996b4bd58f2a92344a26fe78'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.386218 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1451/2920] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:36:06 2022] DELETE /identity/v3/users/8a77b0db996b4bd58f2a92344a26fe78 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:36:06.630164 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:36:06.640421 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf3c97-bf16-407d-a860-5f735d626a49 service neutron] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:36:06.702424 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:36:06.704481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:36:06.704481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:36:06.704481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] PATH_INFO: `/v3/projects/52aad667fe8d4b6582a7e4340ecfe33f` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:36:06.704776 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52aad667fe8d4b6582a7e4340ecfe33f)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:36:06.706609 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:36:06.823082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52aad667fe8d4b6582a7e4340ecfe33f', 'request_id': 'req-6930df5f-6993-409c-b0ac-635ca5b24b91'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.828642 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52aad667fe8d4b6582a7e4340ecfe33f', 'request_id': 'req-6930df5f-6993-409c-b0ac-635ca5b24b91'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.909194 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52aad667fe8d4b6582a7e4340ecfe33f', 'request_id': 'req-6930df5f-6993-409c-b0ac-635ca5b24b91'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.921694 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] The token cache is being invalidate because project 52aad667fe8d4b6582a7e4340ecfe33f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:36:06.922082 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.922716 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:06.924206 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1470/2921] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:36:06 2022] DELETE /identity/v3/projects/52aad667fe8d4b6582a7e4340ecfe33f => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:36:06.928291 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:36:06.929731 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:36:06.942592 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9501a939-d723-4bac-a2a8-6a88cf4c7a5b admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:36:07.000591 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:36:07.001434 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:36:07.001833 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:36:07.002116 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] PATH_INFO: `/v3/users/fb4c7c20d4564097acec780f97479d3d` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:36:07.003561 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb4c7c20d4564097acec780f97479d3d)` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:36:07.005896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:36:07.163282 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb4c7c20d4564097acec780f97479d3d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.175419 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb4c7c20d4564097acec780f97479d3d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.176095 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb4c7c20d4564097acec780f97479d3d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.223841 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb4c7c20d4564097acec780f97479d3d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.226286 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:36:07.231716 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.notifications [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb4c7c20d4564097acec780f97479d3d'} {{(pid=81763) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.237628 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1452/2922] 10.222.0.16 () {58 vars in 1224 bytes} [Thu Aug 25 05:36:06 2022] DELETE /identity/v3/users/fb4c7c20d4564097acec780f97479d3d => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:36:07.487961 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] Authenticating user token {{(pid=81762) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:36:07.489515 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.fernet_utils [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81762) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:36:07.498018 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:36:07.506208 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6930df5f-6993-409c-b0ac-635ca5b24b91 admin admin] Validating token access rules against request {{(pid=81762) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:36:07.563903 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2010f49b97347fb8aff29794f27d93f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f65e42175ca249f08b4d6dc3b72146f4', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81762) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:36:07.564541 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:36:07.564817 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] SCRIPT_NAME: `/identity` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:36:07.565119 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] PATH_INFO: `/v3/projects/a1e042d8df5e4494a03603ae00efd15b` {{(pid=81762) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:36:07.566268 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1e042d8df5e4494a03603ae00efd15b)` {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:36:07.568200 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] RBAC: Authorization granted {{(pid=81762) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:36:07.702617 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1e042d8df5e4494a03603ae00efd15b', 'request_id': 'req-551285e5-b45e-48a2-a671-71e4d377e98f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.708028 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1e042d8df5e4494a03603ae00efd15b', 'request_id': 'req-551285e5-b45e-48a2-a671-71e4d377e98f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.777154 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1e042d8df5e4494a03603ae00efd15b', 'request_id': 'req-551285e5-b45e-48a2-a671-71e4d377e98f'} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.788481 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] The token cache is being invalidate because project a1e042d8df5e4494a03603ae00efd15b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81762) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 25 05:36:07.788773 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.789397 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG keystone.notifications [None req-551285e5-b45e-48a2-a671-71e4d377e98f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81762) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 25 05:36:07.790942 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: [pid: 81762|app: 0|req: 1471/2923] 10.222.0.16 () {58 vars in 1233 bytes} [Thu Aug 25 05:36:07 2022] DELETE /identity/v3/projects/a1e042d8df5e4494a03603ae00efd15b => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 25 05:36:17.537331 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=4 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:36:17.785464 dv-cinder-853910-2-iscsi devstack@keystone.service[81762]: DEBUG dbcounter [-] [81762] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81762) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 25 05:37:51.243192 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] Authenticating user token {{(pid=81763) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 25 05:37:51.244644 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:37:51.313267 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ca982e-efda-44db-bdba-f0fd84f8e90c admin admin] Validating token access rules against request {{(pid=81763) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 25 05:37:51.373750 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0ddac3-751a-470b-97ed-a4f321c13786 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e1ffa92d7fb546399bc404053c201426', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '14d7f359adfb449ab92725f20bfa3774', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81763) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 25 05:37:51.374351 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ddac3-751a-470b-97ed-a4f321c13786 service placement] REQUEST_METHOD: `GET` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 25 05:37:51.374632 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ddac3-751a-470b-97ed-a4f321c13786 service placement] SCRIPT_NAME: `/identity` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 25 05:37:51.374896 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0ddac3-751a-470b-97ed-a4f321c13786 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81763) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 25 05:37:51.376676 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.fernet_utils [None req-5a0ddac3-751a-470b-97ed-a4f321c13786 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81763) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 25 05:37:51.387848 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0ddac3-751a-470b-97ed-a4f321c13786 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 25 05:37:51.389440 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0ddac3-751a-470b-97ed-a4f321c13786 service placement] RBAC: Authorization granted {{(pid=81763) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 25 05:37:51.399270 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: [pid: 81763|app: 0|req: 1453/2924] 10.222.0.16 () {60 vars in 1365 bytes} [Thu Aug 25 05:37:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 25 05:38:01.395040 dv-cinder-853910-2-iscsi devstack@keystone.service[81763]: DEBUG dbcounter [-] [81763] Writing DB stats keystone:SELECT=58 {{(pid=81763) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}